Jul 27 23:15:57.072134 ip-10-0-141-248 systemd[1]: Starting Kubernetes Kubelet... Jul 27 23:15:57.513167 ip-10-0-141-248 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 23:15:57.513167 ip-10-0-141-248 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 23:15:57.513167 ip-10-0-141-248 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 23:15:57.513167 ip-10-0-141-248 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 23:15:57.513167 ip-10-0-141-248 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 23:15:57.515109 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.514958 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.519632 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.520090 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.520431 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.520635 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521655 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521677 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521682 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521686 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521690 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521693 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521697 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521701 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521704 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521707 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521711 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521713 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521716 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521719 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:15:57.526866 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521722 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521725 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521727 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521730 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521732 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521735 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521737 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521740 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521742 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521745 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521747 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521750 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521752 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521755 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521757 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521760 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521766 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521770 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521773 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521776 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:15:57.527389 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521779 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521782 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521785 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521789 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521793 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521796 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521799 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521801 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521804 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521806 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521809 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521812 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521835 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521837 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521840 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521843 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521845 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521848 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521852 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521854 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:15:57.527945 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521857 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521859 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521862 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521865 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521868 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521870 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521873 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521875 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521878 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521881 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521886 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521889 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521892 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521894 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521897 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521900 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521902 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521905 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521909 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521912 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:15:57.528455 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521915 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521917 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521922 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521925 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521928 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521931 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521935 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521937 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521940 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521943 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521946 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521948 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521951 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521953 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521956 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521959 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521962 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521965 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521968 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:15:57.529082 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521970 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.521973 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524339 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524350 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524353 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524357 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524360 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524363 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524367 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524371 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524373 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524376 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524379 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524382 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524384 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524387 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524389 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524391 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:15:57.529535 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524394 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524397 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524399 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524402 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524405 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524407 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524409 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524412 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524414 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524417 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524419 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524421 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524424 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524426 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524430 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524433 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524436 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524439 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524443 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:15:57.530033 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524445 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524448 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524450 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524453 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524455 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524458 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524461 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524463 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524466 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524468 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524470 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524473 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524475 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524477 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524480 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524482 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524485 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524487 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524490 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524492 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524495 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:15:57.530538 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524497 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524500 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524502 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524505 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524507 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524509 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524512 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524514 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524517 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524520 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524522 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524525 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524528 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524532 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524536 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524539 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524541 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524544 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524546 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524548 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:15:57.531065 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524551 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524553 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524556 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524558 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524561 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524563 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524565 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524568 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524571 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524573 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524576 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524578 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524581 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524583 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524585 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524588 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524590 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524592 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524595 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524598 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:15:57.531558 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524600 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524603 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.524606 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525012 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525017 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525020 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525022 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525025 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525028 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525030 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525033 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525036 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525038 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525041 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525043 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525049 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525051 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525054 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525056 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525059 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:15:57.532070 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525062 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525064 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525067 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525069 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525072 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525074 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525077 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525080 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525084 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525088 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525090 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525093 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525096 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525099 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525102 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525104 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525107 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525110 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525113 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:15:57.532570 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525115 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525118 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525121 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525123 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525126 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525128 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525131 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525133 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525137 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525140 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525142 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525146 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525148 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525151 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525153 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525155 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525158 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525160 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525162 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:15:57.533088 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525165 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525168 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525172 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525174 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525177 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525180 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525182 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525186 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525188 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525191 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525193 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525196 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525199 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525201 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525204 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525206 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525209 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525212 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525214 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:15:57.533550 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525217 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525219 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525223 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525226 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525229 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525232 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525235 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525238 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525240 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525243 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525245 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525248 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525250 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525252 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525255 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525257 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525260 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525262 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525264 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525266 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:15:57.534043 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525270 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525272 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525275 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525278 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525280 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525346 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525356 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525363 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525367 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525371 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525374 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525378 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525382 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525386 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525390 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525393 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525397 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525400 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525403 2564 flags.go:64] FLAG: --cgroup-root="" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525405 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525408 2564 flags.go:64] FLAG: --client-ca-file="" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525411 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525414 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 23:15:57.534783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525418 2564 flags.go:64] FLAG: --cluster-domain="" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525421 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525424 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525427 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525430 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525433 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525436 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525439 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525442 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525447 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525450 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525453 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525456 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525459 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525463 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525466 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525469 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525472 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525475 2564 flags.go:64] FLAG: --enable-server="true" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525478 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525483 2564 flags.go:64] FLAG: --event-burst="100" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525486 2564 flags.go:64] FLAG: --event-qps="50" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525488 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525492 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 23:15:57.535590 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525495 2564 flags.go:64] FLAG: --eviction-hard="" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525499 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525502 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525505 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525508 2564 flags.go:64] FLAG: --eviction-soft="" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525510 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525513 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525516 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525519 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525522 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525525 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525527 2564 flags.go:64] FLAG: --feature-gates="" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525531 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525534 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525537 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525540 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525543 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525547 2564 flags.go:64] FLAG: --help="false" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525549 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525552 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525555 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525558 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525561 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 23:15:57.536247 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525564 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525568 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525571 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525574 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525576 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525580 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525582 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525585 2564 flags.go:64] FLAG: --kube-reserved="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525589 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525592 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525595 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525597 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525600 2564 flags.go:64] FLAG: --lock-file="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525603 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525605 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525608 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525614 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525617 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525619 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525622 2564 flags.go:64] FLAG: --logging-format="text" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525625 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525628 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525630 2564 flags.go:64] FLAG: --manifest-url="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525633 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525637 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 23:15:57.536985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525640 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525645 2564 flags.go:64] FLAG: --max-pods="110" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525648 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525651 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525653 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525657 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525659 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525662 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525665 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525673 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525676 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525679 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525682 2564 flags.go:64] FLAG: --pod-cidr="" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525684 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525690 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525694 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525697 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525700 2564 flags.go:64] FLAG: --port="10250" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525703 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525706 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09bb3064d60d597ec" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525708 2564 flags.go:64] FLAG: --qos-reserved="" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525711 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525714 2564 flags.go:64] FLAG: --register-node="true" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525717 2564 flags.go:64] FLAG: --register-with-taints="" Jul 27 23:15:57.537602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525720 2564 flags.go:64] FLAG: --registry-burst="10" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525723 2564 flags.go:64] FLAG: --registry-qps="5" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525726 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525728 2564 flags.go:64] FLAG: --reserved-memory="" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525732 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525735 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525738 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525741 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525743 2564 flags.go:64] FLAG: --runonce="false" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525746 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525751 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525754 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525758 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525761 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525763 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525766 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525769 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525772 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525775 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525778 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525780 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525783 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525786 2564 flags.go:64] FLAG: --system-cgroups="" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525790 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525795 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 23:15:57.538202 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525798 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525801 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525804 2564 flags.go:64] FLAG: --tls-min-version="" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525807 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525809 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525831 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525836 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525839 2564 flags.go:64] FLAG: --v="2" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525844 2564 flags.go:64] FLAG: --version="false" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525847 2564 flags.go:64] FLAG: --vmodule="" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525852 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.525855 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525968 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525973 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525976 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525979 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525982 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525986 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525989 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525991 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525994 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525996 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:15:57.538836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.525999 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526002 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526005 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526007 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526010 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526013 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526015 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526018 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526021 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526024 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526026 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526028 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526031 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526033 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526035 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526038 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526040 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526043 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526045 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526048 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:15:57.539382 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526050 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526052 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526055 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526057 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526060 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526062 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526064 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526068 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526070 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526073 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526075 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526078 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526080 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526083 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526085 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526088 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526090 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526093 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526095 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:15:57.539977 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526098 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526103 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526105 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526108 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526110 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526113 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526115 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526118 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526120 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526123 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526125 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526127 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526130 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526132 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526136 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526140 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526143 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526145 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526147 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526150 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:15:57.540484 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526155 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526158 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526160 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526163 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526165 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526168 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526170 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526173 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526175 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526177 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526180 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526183 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526186 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526190 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526193 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526196 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526199 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526202 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526204 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:15:57.540985 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526207 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526210 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526214 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526217 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526219 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526222 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526224 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526227 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526229 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526232 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.526234 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.527244 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.535908 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.535926 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.537642 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 23:15:57.541438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.537650 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537710 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537717 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537720 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537723 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537726 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537728 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537732 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537734 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537737 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537740 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537742 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537745 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537748 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537750 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537753 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537756 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537759 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537761 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537763 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:15:57.541830 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537766 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537768 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537771 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537773 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537777 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537781 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537784 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537786 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537789 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537792 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537796 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537799 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537801 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537804 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537807 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537809 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537826 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537829 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537832 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537835 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:15:57.542310 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537838 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537841 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537843 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537846 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537849 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537852 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537854 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537858 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537861 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537864 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537867 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537869 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537871 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537874 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537877 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537879 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537900 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537904 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537907 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:15:57.542856 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537909 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537912 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537915 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537917 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537920 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537923 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537926 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537928 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537931 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537933 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537936 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537939 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537941 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537944 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537947 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537949 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537951 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537954 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537956 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537959 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:15:57.543321 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537961 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537965 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537968 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537972 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537975 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537978 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537981 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537984 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537987 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537990 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537992 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537995 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.537997 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538000 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538003 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538006 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538009 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538011 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538014 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:15:57.543864 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538016 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538019 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.538024 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538142 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538146 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538149 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538152 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538155 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538158 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538161 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538163 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538165 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538168 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538172 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538175 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 23:15:57.544335 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538179 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538182 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538184 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538187 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538189 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538192 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538194 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538197 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538199 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538202 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538204 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538207 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538209 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538213 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538216 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538218 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538221 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538223 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538226 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538228 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 23:15:57.544702 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538231 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538234 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538237 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538240 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538243 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538245 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538247 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538250 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538252 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538255 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538258 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538260 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538262 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538266 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538268 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538271 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538273 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538275 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538278 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538280 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 23:15:57.545199 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538283 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538285 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538287 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538290 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538292 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538296 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538300 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538303 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538305 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538308 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538311 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538328 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538331 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538334 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538337 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538340 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538343 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538345 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538348 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 23:15:57.545681 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538350 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538353 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538356 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538358 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538361 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538363 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538365 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538368 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538371 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538373 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538376 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538379 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538381 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538383 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538386 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538388 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538390 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538393 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538398 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538400 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 23:15:57.546153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538403 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538405 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538407 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538410 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538412 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538415 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538417 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:57.538420 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.538424 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.538549 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.542175 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.543620 2564 server.go:1019] "Starting client certificate rotation" Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.543722 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 23:15:57.546634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.543755 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 23:15:57.547003 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.546972 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 23:15:57.548332 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.548319 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 23:15:57.558694 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.558673 2564 log.go:25] "Validated CRI v1 runtime API" Jul 27 23:15:57.561182 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.561166 2564 log.go:25] "Validated CRI v1 image API" Jul 27 23:15:57.561913 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.561898 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 23:15:57.565210 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.565165 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b558a828-2906-4e28-aee5-57777c3dddb9:/dev/nvme0n1p4 c7f73a66-e838-4997-b19a-7ef9be73d33a:/dev/nvme0n1p3] Jul 27 23:15:57.565265 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.565210 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 23:15:57.570052 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.570036 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 23:15:57.570668 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.570562 2564 manager.go:216] Machine: {Timestamp:2026-07-27 23:15:57.569158257 +0000 UTC m=+0.367122873 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102125 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c96ee3f07b5f93cef0157fa896526 SystemUUID:ec2c96ee-3f07-b5f9-3cef-0157fa896526 BootID:a9b78749-7fdd-44e8-a0fe-28a91918aa08 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:59:04:0f:b1:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:59:04:0f:b1:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:4e:86:6e:8c:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 23:15:57.570668 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.570668 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 23:15:57.570761 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.570753 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 23:15:57.574348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.574326 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 23:15:57.574508 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.574351 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 23:15:57.575275 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.575265 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 23:15:57.575306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.575278 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 23:15:57.575306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.575286 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 23:15:57.576125 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.576115 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 23:15:57.577204 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.577195 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 23:15:57.579075 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579063 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 27 23:15:57.579222 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579212 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 23:15:57.579255 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579243 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 27 23:15:57.579255 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579254 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 23:15:57.579306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579269 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 23:15:57.579306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579279 2564 kubelet.go:398] "Adding apiserver pod source" Jul 27 23:15:57.579306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579287 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 23:15:57.579859 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579844 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 23:15:57.579900 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.579873 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 23:15:57.583165 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.583144 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 23:15:57.585535 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.585520 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 23:15:57.585593 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.585552 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 23:15:57.587518 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587507 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587525 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587531 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587536 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587547 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587554 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587562 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587567 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587574 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 23:15:57.587579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587579 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 23:15:57.587851 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587588 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 23:15:57.587851 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.587593 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 23:15:57.589268 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.589258 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 23:15:57.589268 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.589268 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 23:15:57.592435 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.592423 2564 server.go:1268] "Started kubelet" Jul 27 23:15:57.592585 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.592530 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 23:15:57.592585 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.592553 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 23:15:57.592900 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.592881 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 23:15:57.593322 ip-10-0-141-248 systemd[1]: Started Kubernetes Kubelet. Jul 27 23:15:57.594595 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.594582 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 27 23:15:57.597284 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.597263 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 23:15:57.597437 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.597419 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 23:15:57.597510 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.597433 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 23:15:57.597566 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.597505 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 23:15:57.598782 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.597633 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-248.ec2.internal.18c648a0c265baa2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-248.ec2.internal,UID:ip-10-0-141-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-248.ec2.internal,},FirstTimestamp:2026-07-27 23:15:57.592394402 +0000 UTC m=+0.390359017,LastTimestamp:2026-07-27 23:15:57.592394402 +0000 UTC m=+0.390359017,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-248.ec2.internal,}" Jul 27 23:15:57.598883 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.598869 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 23:15:57.605967 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.605952 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 23:15:57.606345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.605957 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 23:15:57.606345 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.606059 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:57.606747 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606734 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 23:15:57.606905 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606750 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 23:15:57.606905 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606765 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 23:15:57.606905 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606843 2564 factory.go:55] Registering systemd factory Jul 27 23:15:57.606905 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606859 2564 factory.go:223] Registration of the systemd container factory successfully Jul 27 23:15:57.607101 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606920 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 27 23:15:57.607101 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.606928 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 27 23:15:57.607169 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.607101 2564 factory.go:153] Registering CRI-O factory Jul 27 23:15:57.607169 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.607113 2564 factory.go:223] Registration of the crio container factory successfully Jul 27 23:15:57.607244 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.607173 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 23:15:57.607244 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.607199 2564 factory.go:103] Registering Raw factory Jul 27 23:15:57.607244 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.607219 2564 manager.go:1196] Started watching for new ooms in manager Jul 27 23:15:57.607673 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.607660 2564 manager.go:349] Starting recovery of all containers Jul 27 23:15:57.607959 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.607938 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 23:15:57.608108 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.608089 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 23:15:57.618518 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.618452 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gsbkl" Jul 27 23:15:57.618619 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.618598 2564 manager.go:354] Recovery completed Jul 27 23:15:57.620133 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.620114 2564 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 27 23:15:57.623158 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.623145 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.625118 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625103 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.625180 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625133 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.625180 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625144 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.625666 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625652 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 23:15:57.625666 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625664 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 23:15:57.625769 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625679 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 27 23:15:57.625769 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625697 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.625853 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.625786 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gsbkl" Jul 27 23:15:57.626502 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.626485 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.626547 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.626518 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.626547 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.626534 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.628336 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.628322 2564 policy_none.go:49] "None policy: Start" Jul 27 23:15:57.628396 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.628343 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 23:15:57.628396 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.628357 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 23:15:57.629538 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.629527 2564 policy_none.go:47] "Start" Jul 27 23:15:57.665506 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665373 2564 manager.go:337] "Starting Device Plugin manager" Jul 27 23:15:57.665614 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.665517 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 23:15:57.665614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665527 2564 server.go:87] "Starting device plugin registration server" Jul 27 23:15:57.665747 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665736 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 23:15:57.665780 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665748 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 23:15:57.665870 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665855 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 23:15:57.665972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665929 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 23:15:57.665972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.665940 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 23:15:57.666506 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.666483 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 23:15:57.666591 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.666543 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:57.679833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.679791 2564 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2597 into the system container \"/system.slice\": " Jul 27 23:15:57.766274 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.766192 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.767546 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.767243 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.767546 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.767273 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.767546 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.767283 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.767546 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.767309 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.774355 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.774328 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 23:15:57.775838 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.775806 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 23:15:57.775921 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.775857 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 23:15:57.775921 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.775878 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 23:15:57.776002 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.775927 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 23:15:57.778011 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.777993 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:15:57.779179 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.779164 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.779272 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.779195 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-248.ec2.internal\": node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:57.805830 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.805787 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:57.876795 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.876734 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal"] Jul 27 23:15:57.876955 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.876880 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.878483 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.878460 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.878598 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.878498 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.878598 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.878513 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.879654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.879641 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.879773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.879747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.879831 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.879782 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.880414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.880396 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.880498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.880425 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.880498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.880428 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.880498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.880451 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.880498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.880461 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.880498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.880435 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.881689 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.881675 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.881755 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.881698 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 23:15:57.882371 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.882355 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientMemory" Jul 27 23:15:57.882430 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.882383 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 23:15:57.882430 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.882393 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeHasSufficientPID" Jul 27 23:15:57.901362 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.901336 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-248.ec2.internal\" not found" node="ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.904827 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.904796 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-248.ec2.internal\" not found" node="ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.906873 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:57.906859 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:57.909067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.909054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf40819b81c5d705f10dd9a192febec3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal\" (UID: \"cf40819b81c5d705f10dd9a192febec3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.909115 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.909075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf40819b81c5d705f10dd9a192febec3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal\" (UID: \"cf40819b81c5d705f10dd9a192febec3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:57.909115 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:57.909092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12d7ac228f243a07d086e226310b600c-config\") pod \"kube-apiserver-proxy-ip-10-0-141-248.ec2.internal\" (UID: \"12d7ac228f243a07d086e226310b600c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.006941 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.006906 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:58.010042 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.010020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf40819b81c5d705f10dd9a192febec3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal\" (UID: \"cf40819b81c5d705f10dd9a192febec3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.010100 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.010050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf40819b81c5d705f10dd9a192febec3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal\" (UID: \"cf40819b81c5d705f10dd9a192febec3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.010100 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.010069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12d7ac228f243a07d086e226310b600c-config\") pod \"kube-apiserver-proxy-ip-10-0-141-248.ec2.internal\" (UID: \"12d7ac228f243a07d086e226310b600c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.010159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.010119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/12d7ac228f243a07d086e226310b600c-config\") pod \"kube-apiserver-proxy-ip-10-0-141-248.ec2.internal\" (UID: \"12d7ac228f243a07d086e226310b600c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.010159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.010129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf40819b81c5d705f10dd9a192febec3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal\" (UID: \"cf40819b81c5d705f10dd9a192febec3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.010218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.010119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf40819b81c5d705f10dd9a192febec3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal\" (UID: \"cf40819b81c5d705f10dd9a192febec3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.107584 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.107529 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:58.203940 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.203903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.207863 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.207670 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:58.208789 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.208775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.308161 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.308123 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:58.408699 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.408641 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:58.509157 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.509125 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-248.ec2.internal\" not found" Jul 27 23:15:58.543454 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.543424 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 23:15:58.543899 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.543572 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 23:15:58.558924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.558903 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:15:58.579464 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.579437 2564 apiserver.go:52] "Watching apiserver" Jul 27 23:15:58.587685 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.587653 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 23:15:58.589457 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.589434 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-4n869","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv","openshift-cluster-node-tuning-operator/tuned-h8fjb","openshift-image-registry/node-ca-wld4r","openshift-multus/multus-z8qh9","openshift-multus/network-metrics-daemon-l6jq8","openshift-network-diagnostics/network-check-target-lmnn5","kube-system/global-pull-secret-syncer-vw4gr","openshift-dns/node-resolver-tlxxm","openshift-multus/multus-additional-cni-plugins-gd2zk","openshift-network-operator/iptables-alerter-4gfkc","openshift-ovn-kubernetes/ovnkube-node-67nrk"] Jul 27 23:15:58.590715 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.590698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.591924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.591902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.593163 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.593122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.593308 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.593275 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 23:15:58.593500 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.593481 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 23:15:58.593585 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.593498 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-st9mz\"" Jul 27 23:15:58.593891 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.593871 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 23:15:58.594120 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.594103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x8hlr\"" Jul 27 23:15:58.594212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.594141 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.594212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.594205 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.594319 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.594238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.594948 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.594929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v7j7m\"" Jul 27 23:15:58.595186 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.595166 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.595283 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.595226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.595449 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.595435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.596042 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.596026 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 23:15:58.596123 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.596047 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4xtv\"" Jul 27 23:15:58.596366 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.596350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.596438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.596395 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.596568 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.596553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:58.596627 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.596606 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:15:58.597043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.597028 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.597292 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.597270 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 23:15:58.597379 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.597314 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.597379 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.597314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98zpz\"" Jul 27 23:15:58.597460 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.597389 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 23:15:58.599691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.598216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:15:58.599691 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.598289 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:15:58.599863 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.599798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.599981 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.599948 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:15:58.601600 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.601581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.602840 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.602808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.603421 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.603406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9fxkk\"" Jul 27 23:15:58.603470 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.603408 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.603470 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.603406 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.604075 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.604058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.604631 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.604611 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 23:15:58.604723 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.604698 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 23:15:58.604809 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.604795 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zwrvv\"" Jul 27 23:15:58.605238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.605223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.605749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.605735 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 23:15:58.606078 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.606065 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.606226 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.606214 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.606429 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.606417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s6qlb\"" Jul 27 23:15:58.606953 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.606932 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 23:15:58.607039 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.606970 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.607039 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.606986 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 23:15:58.607516 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607497 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m5n7w\"" Jul 27 23:15:58.607603 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607546 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 23:15:58.607603 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607568 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 23:15:58.607603 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607583 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 23:15:58.607773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607505 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 23:15:58.607773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607770 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 23:15:58.607888 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.607864 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 23:15:58.612455 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-netns\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.612561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-conf-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.612561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-daemon-config\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.612561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-sys\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.612561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-cni-netd\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.612561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-modprobe-d\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgwxv\" (UniqueName: \"kubernetes.io/projected/d92fda41-1b32-4f06-97c5-0d509c87bb90-kube-api-access-zgwxv\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-slash\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-systemd\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-env-overrides\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovnkube-script-lib\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff37ceb7-f0de-44f8-82a6-c34d3899c294-serviceca\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-var-lib-kubelet\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovnkube-config\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.612800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzsw6\" (UniqueName: \"kubernetes.io/projected/ff37ceb7-f0de-44f8-82a6-c34d3899c294-kube-api-access-nzsw6\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-cni-bin\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-multus-certs\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfn97\" (UniqueName: \"kubernetes.io/projected/103dec8d-2e5e-4973-b634-b28b043cfb6b-kube-api-access-bfn97\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9f4s\" (UniqueName: \"kubernetes.io/projected/b9f9724e-dafa-467a-aafe-61b2a5666e86-kube-api-access-p9f4s\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a-konnectivity-ca\") pod \"konnectivity-agent-4n869\" (UID: \"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a\") " pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.612978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-socket-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-system-cni-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d301bf01-44e5-43fa-b2d9-8fccf338c712-cni-binary-copy\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65b91810-92da-4848-b465-2025a4ce7ec0-dbus\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-systemd-units\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-host\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-os-release\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4pm5\" (UniqueName: \"kubernetes.io/projected/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-kube-api-access-t4pm5\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-kubelet\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.613312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-cni-bin\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-k8s-cni-cncf-io\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysctl-d\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b21f0f-0195-4724-8c08-5f0327dd78c0-tmp\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8q7\" (UniqueName: \"kubernetes.io/projected/e8b21f0f-0195-4724-8c08-5f0327dd78c0-kube-api-access-xt8q7\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-registration-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-etc-kubernetes\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hd7l\" (UniqueName: \"kubernetes.io/projected/d301bf01-44e5-43fa-b2d9-8fccf338c712-kube-api-access-7hd7l\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-kubernetes\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-ovn\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-sys-fs\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpjs\" (UniqueName: \"kubernetes.io/projected/059a60c6-94e4-47a3-b18e-5c347039a27a-kube-api-access-tfpjs\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/103dec8d-2e5e-4973-b634-b28b043cfb6b-hosts-file\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.614375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-etc-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-node-log\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-cni-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-hostroot\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613681 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-lib-modules\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-etc-selinux\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-kubelet\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dwj\" (UniqueName: \"kubernetes.io/projected/93e62be3-2c13-473b-b298-ae8f2dde91b8-kube-api-access-n8dwj\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-os-release\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-iptables-alerter-script\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovn-node-metrics-cert\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.613991 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a-agent-certs\") pod \"konnectivity-agent-4n869\" (UID: \"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a\") " pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65b91810-92da-4848-b465-2025a4ce7ec0-kubelet-config\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103dec8d-2e5e-4973-b634-b28b043cfb6b-tmp-dir\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.614924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-systemd\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-tuned\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-host-slash\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-var-lib-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-run-netns\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-run-ovn-kubernetes\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff37ceb7-f0de-44f8-82a6-c34d3899c294-host\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-cni-multus\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-device-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysctl-conf\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-run\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-socket-dir-parent\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysconfig\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.615414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-cnibin\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.615906 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.615906 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-log-socket\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.615906 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.614494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-cnibin\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.616638 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.616620 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 23:15:58.616746 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.616691 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" Jul 27 23:15:58.616746 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.616723 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal"] Jul 27 23:15:58.618551 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.618536 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 23:15:58.628365 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.628348 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 23:15:58.628623 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.628607 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal"] Jul 27 23:15:58.631638 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.631603 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 23:10:57 +0000 UTC" deadline="2028-03-31 22:56:19.175580987 +0000 UTC" Jul 27 23:15:58.631638 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.631636 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14711h40m20.543947382s" Jul 27 23:15:58.638270 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.638253 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bs7b7" Jul 27 23:15:58.645260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.645241 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:15:58.645703 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.645686 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bs7b7" Jul 27 23:15:58.715624 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4pm5\" (UniqueName: \"kubernetes.io/projected/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-kube-api-access-t4pm5\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-kubelet\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-cni-bin\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-k8s-cni-cncf-io\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysctl-d\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-kubelet\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b21f0f-0195-4724-8c08-5f0327dd78c0-tmp\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-cni-bin\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.715788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8q7\" (UniqueName: \"kubernetes.io/projected/e8b21f0f-0195-4724-8c08-5f0327dd78c0-kube-api-access-xt8q7\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysctl-d\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-registration-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-etc-kubernetes\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-k8s-cni-cncf-io\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hd7l\" (UniqueName: \"kubernetes.io/projected/d301bf01-44e5-43fa-b2d9-8fccf338c712-kube-api-access-7hd7l\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-registration-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-kubernetes\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-etc-kubernetes\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.715999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-ovn\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-sys-fs\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716072 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-kubernetes\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-ovn\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpjs\" (UniqueName: \"kubernetes.io/projected/059a60c6-94e4-47a3-b18e-5c347039a27a-kube-api-access-tfpjs\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/103dec8d-2e5e-4973-b634-b28b043cfb6b-hosts-file\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.716295 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-sys-fs\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-system-cni-dir\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-etc-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-etc-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-node-log\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-cni-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-hostroot\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-lib-modules\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-etc-selinux\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-kubelet\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dwj\" (UniqueName: \"kubernetes.io/projected/93e62be3-2c13-473b-b298-ae8f2dde91b8-kube-api-access-n8dwj\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-os-release\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-hostroot\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-cni-binary-copy\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-node-log\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-iptables-alerter-script\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/103dec8d-2e5e-4973-b634-b28b043cfb6b-hosts-file\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovn-node-metrics-cert\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716615 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-cni-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-etc-selinux\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716837 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-kubelet\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-lib-modules\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.716962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-os-release\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a-agent-certs\") pod \"konnectivity-agent-4n869\" (UID: \"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a\") " pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65b91810-92da-4848-b465-2025a4ce7ec0-kubelet-config\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103dec8d-2e5e-4973-b634-b28b043cfb6b-tmp-dir\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-systemd\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-tuned\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-host-slash\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.717966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-var-lib-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717270 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-run-netns\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-run-ovn-kubernetes\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff37ceb7-f0de-44f8-82a6-c34d3899c294-host\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-cni-multus\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-iptables-alerter-script\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-device-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysctl-conf\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-run\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-var-lib-openvswitch\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-socket-dir-parent\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-device-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysconfig\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-cnibin\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.718737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65b91810-92da-4848-b465-2025a4ce7ec0-kubelet-config\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.717676 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysctl-conf\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.717739 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:15:59.217718171 +0000 UTC m=+2.015682800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-run\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-run-netns\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-run-ovn-kubernetes\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-socket-dir-parent\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ff37ceb7-f0de-44f8-82a6-c34d3899c294-host\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-sysconfig\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-cni-multus\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-cnibin\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-log-socket\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.717995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-cnibin\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-netns\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.719561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-conf-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-daemon-config\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-sys\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-cni-netd\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-modprobe-d\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/103dec8d-2e5e-4973-b634-b28b043cfb6b-tmp-dir\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgwxv\" (UniqueName: \"kubernetes.io/projected/d92fda41-1b32-4f06-97c5-0d509c87bb90-kube-api-access-zgwxv\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-slash\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-systemd\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-env-overrides\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovnkube-script-lib\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff37ceb7-f0de-44f8-82a6-c34d3899c294-serviceca\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-var-lib-kubelet\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovnkube-config\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsw6\" (UniqueName: \"kubernetes.io/projected/ff37ceb7-f0de-44f8-82a6-c34d3899c294-kube-api-access-nzsw6\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-cni-bin\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.720412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-multus-certs\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfn97\" (UniqueName: \"kubernetes.io/projected/103dec8d-2e5e-4973-b634-b28b043cfb6b-kube-api-access-bfn97\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9f4s\" (UniqueName: \"kubernetes.io/projected/b9f9724e-dafa-467a-aafe-61b2a5666e86-kube-api-access-p9f4s\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a-konnectivity-ca\") pod \"konnectivity-agent-4n869\" (UID: \"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a\") " pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-socket-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-system-cni-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d301bf01-44e5-43fa-b2d9-8fccf338c712-cni-binary-copy\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65b91810-92da-4848-b465-2025a4ce7ec0-dbus\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-systemd-units\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-host\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-os-release\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-host-slash\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-env-overrides\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d92fda41-1b32-4f06-97c5-0d509c87bb90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719009 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-sys\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-cni-netd\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719311 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ff37ceb7-f0de-44f8-82a6-c34d3899c294-serviceca\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.719357 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:15:58.721345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-var-lib-cni-bin\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-multus-certs\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-modprobe-d\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-os-release\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719644 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a-konnectivity-ca\") pod \"konnectivity-agent-4n869\" (UID: \"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a\") " pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.719675 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:15:59.219657153 +0000 UTC m=+2.017621770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-var-lib-kubelet\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.719741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/059a60c6-94e4-47a3-b18e-5c347039a27a-socket-dir\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.718792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-host-slash\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b21f0f-0195-4724-8c08-5f0327dd78c0-tmp\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-daemon-config\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovnkube-config\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65b91810-92da-4848-b465-2025a4ce7ec0-dbus\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-system-cni-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-systemd-units\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-host\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-systemd\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.722238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720428 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-host-run-netns\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-log-socket\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-cnibin\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9f9724e-dafa-467a-aafe-61b2a5666e86-run-systemd\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720561 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovnkube-script-lib\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b21f0f-0195-4724-8c08-5f0327dd78c0-etc-tuned\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d301bf01-44e5-43fa-b2d9-8fccf338c712-multus-conf-dir\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a-agent-certs\") pod \"konnectivity-agent-4n869\" (UID: \"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a\") " pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.720691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d301bf01-44e5-43fa-b2d9-8fccf338c712-cni-binary-copy\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.722405 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.722422 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.722435 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:15:58.723325 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:58.722492 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:15:59.222476365 +0000 UTC m=+2.020440969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:15:58.724360 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.724315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d92fda41-1b32-4f06-97c5-0d509c87bb90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.724972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.724867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4pm5\" (UniqueName: \"kubernetes.io/projected/bda620fd-c94b-4f69-a37e-3a4384cd6ad9-kube-api-access-t4pm5\") pod \"iptables-alerter-4gfkc\" (UID: \"bda620fd-c94b-4f69-a37e-3a4384cd6ad9\") " pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.724972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.724923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dwj\" (UniqueName: \"kubernetes.io/projected/93e62be3-2c13-473b-b298-ae8f2dde91b8-kube-api-access-n8dwj\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:58.726060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.725678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9f9724e-dafa-467a-aafe-61b2a5666e86-ovn-node-metrics-cert\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.726060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.726053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8q7\" (UniqueName: \"kubernetes.io/projected/e8b21f0f-0195-4724-8c08-5f0327dd78c0-kube-api-access-xt8q7\") pod \"tuned-h8fjb\" (UID: \"e8b21f0f-0195-4724-8c08-5f0327dd78c0\") " pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.726357 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.726338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hd7l\" (UniqueName: \"kubernetes.io/projected/d301bf01-44e5-43fa-b2d9-8fccf338c712-kube-api-access-7hd7l\") pod \"multus-z8qh9\" (UID: \"d301bf01-44e5-43fa-b2d9-8fccf338c712\") " pod="openshift-multus/multus-z8qh9" Jul 27 23:15:58.727209 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.727051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpjs\" (UniqueName: \"kubernetes.io/projected/059a60c6-94e4-47a3-b18e-5c347039a27a-kube-api-access-tfpjs\") pod \"aws-ebs-csi-driver-node-5lpnv\" (UID: \"059a60c6-94e4-47a3-b18e-5c347039a27a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.727209 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.727114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgwxv\" (UniqueName: \"kubernetes.io/projected/d92fda41-1b32-4f06-97c5-0d509c87bb90-kube-api-access-zgwxv\") pod \"multus-additional-cni-plugins-gd2zk\" (UID: \"d92fda41-1b32-4f06-97c5-0d509c87bb90\") " pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.727880 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.727862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsw6\" (UniqueName: \"kubernetes.io/projected/ff37ceb7-f0de-44f8-82a6-c34d3899c294-kube-api-access-nzsw6\") pod \"node-ca-wld4r\" (UID: \"ff37ceb7-f0de-44f8-82a6-c34d3899c294\") " pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.728041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.728023 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfn97\" (UniqueName: \"kubernetes.io/projected/103dec8d-2e5e-4973-b634-b28b043cfb6b-kube-api-access-bfn97\") pod \"node-resolver-tlxxm\" (UID: \"103dec8d-2e5e-4973-b634-b28b043cfb6b\") " pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:58.728679 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.728656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9f4s\" (UniqueName: \"kubernetes.io/projected/b9f9724e-dafa-467a-aafe-61b2a5666e86-kube-api-access-p9f4s\") pod \"ovnkube-node-67nrk\" (UID: \"b9f9724e-dafa-467a-aafe-61b2a5666e86\") " pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.729201 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.729181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" Jul 27 23:15:58.732511 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.732497 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4gfkc" Jul 27 23:15:58.734113 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.734085 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf40819b81c5d705f10dd9a192febec3.slice/crio-7a709096ab3108ebb9ecbc59f6d64c6b0b74f5e46f898da8d3816a296adf17cb WatchSource:0}: Error finding container 7a709096ab3108ebb9ecbc59f6d64c6b0b74f5e46f898da8d3816a296adf17cb: Status 404 returned error can't find the container with id 7a709096ab3108ebb9ecbc59f6d64c6b0b74f5e46f898da8d3816a296adf17cb Jul 27 23:15:58.734710 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.734679 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d7ac228f243a07d086e226310b600c.slice/crio-ab6c491b1ec21b179e5d14ecf3d21e4181f9c922502293da38aaaa415f57466a WatchSource:0}: Error finding container ab6c491b1ec21b179e5d14ecf3d21e4181f9c922502293da38aaaa415f57466a: Status 404 returned error can't find the container with id ab6c491b1ec21b179e5d14ecf3d21e4181f9c922502293da38aaaa415f57466a Jul 27 23:15:58.741883 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.740444 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 23:15:58.744670 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.744644 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda620fd_c94b_4f69_a37e_3a4384cd6ad9.slice/crio-06ec7d8fdc1261caf2157659678ffaf851fbd0dfb3336d776dce90cd524778f0 WatchSource:0}: Error finding container 06ec7d8fdc1261caf2157659678ffaf851fbd0dfb3336d776dce90cd524778f0: Status 404 returned error can't find the container with id 06ec7d8fdc1261caf2157659678ffaf851fbd0dfb3336d776dce90cd524778f0 Jul 27 23:15:58.779047 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.778993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4gfkc" event={"ID":"bda620fd-c94b-4f69-a37e-3a4384cd6ad9","Type":"ContainerStarted","Data":"06ec7d8fdc1261caf2157659678ffaf851fbd0dfb3336d776dce90cd524778f0"} Jul 27 23:15:58.780007 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.779987 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerStarted","Data":"e62f08415e924790fb901f40a6c81c5a10a63c2e0df443cfb441ff206ad7fd28"} Jul 27 23:15:58.781053 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.781020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" event={"ID":"12d7ac228f243a07d086e226310b600c","Type":"ContainerStarted","Data":"ab6c491b1ec21b179e5d14ecf3d21e4181f9c922502293da38aaaa415f57466a"} Jul 27 23:15:58.782010 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.781990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" event={"ID":"cf40819b81c5d705f10dd9a192febec3","Type":"ContainerStarted","Data":"7a709096ab3108ebb9ecbc59f6d64c6b0b74f5e46f898da8d3816a296adf17cb"} Jul 27 23:15:58.813681 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.813653 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:15:58.925967 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.925896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4n869" Jul 27 23:15:58.931326 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.931308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:15:58.935931 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.935907 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5455e7ab_9da8_45ac_8d9a_ea1f93ac7a0a.slice/crio-d82c92314504db48f8ef49e92ec0b1250aad9c73f4b557dd49b4e07949e393f1 WatchSource:0}: Error finding container d82c92314504db48f8ef49e92ec0b1250aad9c73f4b557dd49b4e07949e393f1: Status 404 returned error can't find the container with id d82c92314504db48f8ef49e92ec0b1250aad9c73f4b557dd49b4e07949e393f1 Jul 27 23:15:58.940153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.940120 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f9724e_dafa_467a_aafe_61b2a5666e86.slice/crio-e75933c81a971801f63843a858c9dfc5a3339354d64b19e0fa0e2b02d5474a24 WatchSource:0}: Error finding container e75933c81a971801f63843a858c9dfc5a3339354d64b19e0fa0e2b02d5474a24: Status 404 returned error can't find the container with id e75933c81a971801f63843a858c9dfc5a3339354d64b19e0fa0e2b02d5474a24 Jul 27 23:15:58.956931 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.956910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" Jul 27 23:15:58.963721 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.963697 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059a60c6_94e4_47a3_b18e_5c347039a27a.slice/crio-ef4ac857de3239512ebe7959c72472f4a537b57317ebff736d54f60c72402183 WatchSource:0}: Error finding container ef4ac857de3239512ebe7959c72472f4a537b57317ebff736d54f60c72402183: Status 404 returned error can't find the container with id ef4ac857de3239512ebe7959c72472f4a537b57317ebff736d54f60c72402183 Jul 27 23:15:58.969249 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.969233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" Jul 27 23:15:58.975363 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.975343 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b21f0f_0195_4724_8c08_5f0327dd78c0.slice/crio-25162339d889bb20a34bbe1d5cdfaad75ccd68e93e98772041711120530a3bc3 WatchSource:0}: Error finding container 25162339d889bb20a34bbe1d5cdfaad75ccd68e93e98772041711120530a3bc3: Status 404 returned error can't find the container with id 25162339d889bb20a34bbe1d5cdfaad75ccd68e93e98772041711120530a3bc3 Jul 27 23:15:58.984999 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:58.984983 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wld4r" Jul 27 23:15:58.991868 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:58.991850 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff37ceb7_f0de_44f8_82a6_c34d3899c294.slice/crio-9c56732e09729243ff89dca4ef7e0e0d0506c47808e182659d7e5cffc3f43d87 WatchSource:0}: Error finding container 9c56732e09729243ff89dca4ef7e0e0d0506c47808e182659d7e5cffc3f43d87: Status 404 returned error can't find the container with id 9c56732e09729243ff89dca4ef7e0e0d0506c47808e182659d7e5cffc3f43d87 Jul 27 23:15:59.002442 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.002426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z8qh9" Jul 27 23:15:59.010402 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:59.010381 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd301bf01_44e5_43fa_b2d9_8fccf338c712.slice/crio-927ee38150c85b2ad37b098a422d269c3079250994351862198f6089e8c6fbdd WatchSource:0}: Error finding container 927ee38150c85b2ad37b098a422d269c3079250994351862198f6089e8c6fbdd: Status 404 returned error can't find the container with id 927ee38150c85b2ad37b098a422d269c3079250994351862198f6089e8c6fbdd Jul 27 23:15:59.019218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.019198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlxxm" Jul 27 23:15:59.027123 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:15:59.027101 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103dec8d_2e5e_4973_b634_b28b043cfb6b.slice/crio-677d596cb6d1be4eca0bcf2d32a47be42c3c6cc6a26fde11220a26284714bc61 WatchSource:0}: Error finding container 677d596cb6d1be4eca0bcf2d32a47be42c3c6cc6a26fde11220a26284714bc61: Status 404 returned error can't find the container with id 677d596cb6d1be4eca0bcf2d32a47be42c3c6cc6a26fde11220a26284714bc61 Jul 27 23:15:59.223219 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.223132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:15:59.223219 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.223173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:15:59.223219 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.223214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:15:59.223551 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223346 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:15:59.223551 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223406 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:00.223384798 +0000 UTC m=+3.021349404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:15:59.223551 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223518 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:15:59.223551 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223534 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:15:59.223551 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223546 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:15:59.223812 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223588 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:00.223575111 +0000 UTC m=+3.021539716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:15:59.223812 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223650 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:15:59.223812 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.223684 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:00.223673435 +0000 UTC m=+3.021638042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:15:59.647861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.647671 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:10:58 +0000 UTC" deadline="2028-03-28 21:54:46.867286831 +0000 UTC" Jul 27 23:15:59.647861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.647737 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h38m47.21955389s" Jul 27 23:15:59.778918 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.777078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:15:59.778918 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:15:59.777212 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:15:59.803750 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.803708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wld4r" event={"ID":"ff37ceb7-f0de-44f8-82a6-c34d3899c294","Type":"ContainerStarted","Data":"9c56732e09729243ff89dca4ef7e0e0d0506c47808e182659d7e5cffc3f43d87"} Jul 27 23:15:59.816097 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.816056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" event={"ID":"059a60c6-94e4-47a3-b18e-5c347039a27a","Type":"ContainerStarted","Data":"ef4ac857de3239512ebe7959c72472f4a537b57317ebff736d54f60c72402183"} Jul 27 23:15:59.830599 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.830557 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z8qh9" event={"ID":"d301bf01-44e5-43fa-b2d9-8fccf338c712","Type":"ContainerStarted","Data":"927ee38150c85b2ad37b098a422d269c3079250994351862198f6089e8c6fbdd"} Jul 27 23:15:59.846930 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.843789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" event={"ID":"e8b21f0f-0195-4724-8c08-5f0327dd78c0","Type":"ContainerStarted","Data":"25162339d889bb20a34bbe1d5cdfaad75ccd68e93e98772041711120530a3bc3"} Jul 27 23:15:59.857217 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.857179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"e75933c81a971801f63843a858c9dfc5a3339354d64b19e0fa0e2b02d5474a24"} Jul 27 23:15:59.868538 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.868498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4n869" event={"ID":"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a","Type":"ContainerStarted","Data":"d82c92314504db48f8ef49e92ec0b1250aad9c73f4b557dd49b4e07949e393f1"} Jul 27 23:15:59.893340 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:15:59.893204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlxxm" event={"ID":"103dec8d-2e5e-4973-b634-b28b043cfb6b","Type":"ContainerStarted","Data":"677d596cb6d1be4eca0bcf2d32a47be42c3c6cc6a26fde11220a26284714bc61"} Jul 27 23:16:00.048627 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.048543 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 23:16:00.232384 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.232340 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:00.232560 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.232398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:00.232560 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.232444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:00.232692 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232595 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:00.232692 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232660 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:02.232641075 +0000 UTC m=+5.030605681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:00.232813 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232748 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:16:00.232813 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232766 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:16:00.232813 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232778 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:00.233007 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232823 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:02.232811574 +0000 UTC m=+5.030776178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:00.233007 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232904 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:00.233007 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.232941 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:02.232928054 +0000 UTC m=+5.030892663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:00.648562 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.648480 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 23:10:58 +0000 UTC" deadline="2028-03-22 13:09:51.857643709 +0000 UTC" Jul 27 23:16:00.648562 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.648524 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14485h53m51.209125448s" Jul 27 23:16:00.776629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.776594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:00.776869 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.776737 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:00.777218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:00.777196 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:00.777345 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:00.777319 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:01.779750 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:01.779651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:01.780232 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:01.779790 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:02.252275 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:02.252184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:02.252275 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:02.252240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:02.252491 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:02.252281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:02.252491 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.252432 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:02.252611 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.252495 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:06.252473926 +0000 UTC m=+9.050438531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:02.252933 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.252915 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:16:02.253032 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.252937 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:16:02.253032 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.252949 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:02.253032 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.252989 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:06.252977673 +0000 UTC m=+9.050942275 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:02.253193 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.253046 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:02.253193 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.253080 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:06.253069482 +0000 UTC m=+9.051034086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:02.776338 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:02.776303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:02.776523 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:02.776303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:02.776523 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.776445 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:02.776634 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:02.776539 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:03.777305 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:03.776832 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:03.777305 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:03.776973 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:04.776832 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:04.776795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:04.777017 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:04.776926 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:04.777324 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:04.777295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:04.777673 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:04.777400 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:05.780061 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:05.779423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:05.780061 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:05.779549 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:06.283894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:06.283853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:06.284134 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:06.283904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:06.284134 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:06.283948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:06.284134 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284101 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:06.284284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284184 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:14.284163077 +0000 UTC m=+17.082127694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:06.284591 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284564 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:16:06.284591 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284591 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:16:06.284741 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284602 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:06.284741 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284640 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:14.284627076 +0000 UTC m=+17.082591679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:06.284741 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284684 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:06.284741 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.284709 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:14.284703053 +0000 UTC m=+17.082667655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:06.777419 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:06.776898 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:06.777419 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:06.776947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:06.777419 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.777060 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:06.777419 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:06.777199 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:07.778016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:07.777985 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:07.778366 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:07.778100 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:08.777050 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:08.777011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:08.777210 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:08.777168 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:08.777309 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:08.777011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:08.777436 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:08.777410 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:09.776944 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:09.776908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:09.777391 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:09.777037 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:10.776975 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:10.776909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:10.776975 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:10.776937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:10.777361 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:10.777018 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:10.777361 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:10.777108 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:11.777217 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:11.777183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:11.777673 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:11.777315 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:12.776412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:12.776374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:12.776606 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:12.776386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:12.776606 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:12.776487 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:12.776606 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:12.776597 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:13.777097 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:13.777063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:13.777561 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:13.777201 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:14.342040 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:14.342004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:14.342040 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:14.342047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342156 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342183 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342207 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342211 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:30.342194111 +0000 UTC m=+33.140158732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342219 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:14.342251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:14.342284 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342260 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:16:30.342247829 +0000 UTC m=+33.140212451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:14.342609 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342318 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:14.342609 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.342350 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:30.342341235 +0000 UTC m=+33.140305837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:14.777161 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:14.776869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:14.777704 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:14.776927 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:14.777704 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.777222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:14.777704 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:14.777331 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:15.776935 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:15.776900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:15.777120 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:15.777015 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:16.776677 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:16.776647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:16.777159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:16.776645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:16.777159 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:16.776758 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:16.777159 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:16.776854 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:17.778025 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:17.777991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:17.778477 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:17.778106 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:18.777098 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:18.777048 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:18.777098 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:18.777081 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:18.777208 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:18.777133 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:18.777276 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:18.777257 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:19.776677 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.776517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:19.777267 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:19.776756 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:19.943791 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.943758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z8qh9" event={"ID":"d301bf01-44e5-43fa-b2d9-8fccf338c712","Type":"ContainerStarted","Data":"54516aa5db91c8190c0db998af6557192d395cc007104482184dbe8f24140b07"} Jul 27 23:16:19.945218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.945194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" event={"ID":"e8b21f0f-0195-4724-8c08-5f0327dd78c0","Type":"ContainerStarted","Data":"508245fba1240fab0236d34cdb98fe3a3c88e354705f3677e79831fb8c7f8e72"} Jul 27 23:16:19.947867 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.947844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"87d186999d3c0fd6a03b4258fe868f452a3db264c8d787e48bf533790626b23f"} Jul 27 23:16:19.947942 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.947901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"cc51031e193b936774756d9d5c74844c87a98fb68ec39dc37a337b432881a5cc"} Jul 27 23:16:19.947942 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.947935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"669233da5496139bef7ed3e0b739e927556add4d822e9a190c3b0c5ad103fc75"} Jul 27 23:16:19.948014 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.947974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"f1b782e19eb7f3ac09715db7988c1c1820e2a7be0cb418bf4d187d208a429535"} Jul 27 23:16:19.948014 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.948009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"4eedb459e6bc463ccc4f545cf9cf98060e6f0c865c826f43a8ebb7f0b7dc7757"} Jul 27 23:16:19.949170 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.949148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4n869" event={"ID":"5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a","Type":"ContainerStarted","Data":"239972df9f184e78b151a3d7ca72e551bd0eb8f81e1d659167919f3821d8a0bb"} Jul 27 23:16:19.950901 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.950868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" event={"ID":"12d7ac228f243a07d086e226310b600c","Type":"ContainerStarted","Data":"770fb738d54c074226ba7a77485d51b1035215b640309c2c96396ec179655bf4"} Jul 27 23:16:19.954787 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.954758 2564 generic.go:356] "Generic (PLEG): container finished" podID="cf40819b81c5d705f10dd9a192febec3" containerID="92ccb3a2c8284949411971d0aeea02f4b6a58712958fa4211d95b8fa2906f17f" exitCode=0 Jul 27 23:16:19.954878 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.954839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" event={"ID":"cf40819b81c5d705f10dd9a192febec3","Type":"ContainerDied","Data":"92ccb3a2c8284949411971d0aeea02f4b6a58712958fa4211d95b8fa2906f17f"} Jul 27 23:16:19.956154 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.956121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlxxm" event={"ID":"103dec8d-2e5e-4973-b634-b28b043cfb6b","Type":"ContainerStarted","Data":"25c9788b037ffb2695ad8fa8b46ead7c718413508a8dc999f728049701084ece"} Jul 27 23:16:19.957431 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.957410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wld4r" event={"ID":"ff37ceb7-f0de-44f8-82a6-c34d3899c294","Type":"ContainerStarted","Data":"7f40e0d0d18edaf0fa433976af5032ecf2aebae3814eda9a615267fbe52d57c3"} Jul 27 23:16:19.958783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.958754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" event={"ID":"059a60c6-94e4-47a3-b18e-5c347039a27a","Type":"ContainerStarted","Data":"10fdd7d2988a40b9cf3b40730a86d5dfc392af65561c2ad86da159e308f203ee"} Jul 27 23:16:19.959483 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.959448 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z8qh9" podStartSLOduration=1.994907032 podStartE2EDuration="21.959436099s" totalImagesPullingTime="19.964529067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:19.958723667 +0000 UTC m=+22.756688292" watchObservedRunningTime="2026-07-27 23:16:19.959436099 +0000 UTC m=+22.757400724" Jul 27 23:16:19.960252 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.960232 2564 generic.go:356] "Generic (PLEG): container finished" podID="d92fda41-1b32-4f06-97c5-0d509c87bb90" containerID="e4d443a62e595654e731b2c4f09fd669a61a96644e6457a87256e82838b82d07" exitCode=0 Jul 27 23:16:19.960339 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.960258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerDied","Data":"e4d443a62e595654e731b2c4f09fd669a61a96644e6457a87256e82838b82d07"} Jul 27 23:16:19.973538 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.973502 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8fjb" podStartSLOduration=1.97643886 podStartE2EDuration="21.973491348s" totalImagesPullingTime="19.997052488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:19.973259099 +0000 UTC m=+22.771223724" watchObservedRunningTime="2026-07-27 23:16:19.973491348 +0000 UTC m=+22.771455972" Jul 27 23:16:19.985495 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:19.985463 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4n869" podStartSLOduration=6.424232334 podStartE2EDuration="21.985452485s" totalImagesPullingTime="15.561220151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:19.985134334 +0000 UTC m=+22.783098952" watchObservedRunningTime="2026-07-27 23:16:19.985452485 +0000 UTC m=+22.783417108" Jul 27 23:16:20.003103 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.003066 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wld4r" podStartSLOduration=2.498347249 podStartE2EDuration="22.003056366s" totalImagesPullingTime="19.504709117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:20.002838126 +0000 UTC m=+22.800802741" watchObservedRunningTime="2026-07-27 23:16:20.003056366 +0000 UTC m=+22.801021006" Jul 27 23:16:20.019789 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.019756 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tlxxm" podStartSLOduration=2.550205597 podStartE2EDuration="22.019743583s" totalImagesPullingTime="19.469537986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:20.019611625 +0000 UTC m=+22.817576259" watchObservedRunningTime="2026-07-27 23:16:20.019743583 +0000 UTC m=+22.817708207" Jul 27 23:16:20.071919 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.071733 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-248.ec2.internal" podStartSLOduration=22.071719554 podStartE2EDuration="22.071719554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:20.052893233 +0000 UTC m=+22.850857858" watchObservedRunningTime="2026-07-27 23:16:20.071719554 +0000 UTC m=+22.869684179" Jul 27 23:16:20.777140 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.777067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:20.777622 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:20.777202 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:20.777622 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.777274 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:20.777622 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:20.777413 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:20.965587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.965553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"77d31b3466d1d250af01316917361dad041ba4ce1820b2b9548620d28bcbd25c"} Jul 27 23:16:20.967406 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.967382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" event={"ID":"cf40819b81c5d705f10dd9a192febec3","Type":"ContainerStarted","Data":"7989849b727d069fc8a878e6674a5a7ebd72a87928e8be2ab30e43f41b4bb4d8"} Jul 27 23:16:20.968895 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.968862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4gfkc" event={"ID":"bda620fd-c94b-4f69-a37e-3a4384cd6ad9","Type":"ContainerStarted","Data":"d2c83b02062e67bc7cebd381cce3dcaf9a60bcd4b2995cb4c34c3700b35c45ee"} Jul 27 23:16:20.998173 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.998115 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4gfkc" podStartSLOduration=2.772278879 podStartE2EDuration="22.998095433s" totalImagesPullingTime="20.225816554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:20.985109115 +0000 UTC m=+23.783073739" watchObservedRunningTime="2026-07-27 23:16:20.998095433 +0000 UTC m=+23.796060057" Jul 27 23:16:20.998363 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:20.998335 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-248.ec2.internal" podStartSLOduration=22.998327984 podStartE2EDuration="22.998327984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:20.997843491 +0000 UTC m=+23.795808115" watchObservedRunningTime="2026-07-27 23:16:20.998327984 +0000 UTC m=+23.796292611" Jul 27 23:16:21.038508 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.038483 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4n869" Jul 27 23:16:21.039092 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.039072 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4n869" Jul 27 23:16:21.093797 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.093772 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 23:16:21.679696 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.679580 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T23:16:21.093794414Z","UUID":"35c6208c-0651-40fe-89e7-504a02857cd3","Handler":null,"Name":"","Endpoint":""} Jul 27 23:16:21.681660 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.681629 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 23:16:21.681789 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.681695 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 23:16:21.779077 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.779045 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:21.779489 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:21.779140 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:21.973941 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.973847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" event={"ID":"059a60c6-94e4-47a3-b18e-5c347039a27a","Type":"ContainerStarted","Data":"1a718d55faae98bde5abaa4aba991428ab0e9ca4fde708ca71a07350ee8f8c0f"} Jul 27 23:16:21.974436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.974379 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4n869" Jul 27 23:16:21.974646 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:21.974625 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4n869" Jul 27 23:16:22.776050 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:22.776016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:22.776050 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:22.776028 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:22.776285 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:22.776172 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:22.776285 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:22.776185 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:22.980012 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:22.979970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"c874eed89b11eb9984c373caabdf0ff71cc6b14fd2607c0ee3aaea574407bab0"} Jul 27 23:16:23.779946 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:23.779912 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:23.780153 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:23.780024 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:24.777162 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.776999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:24.777744 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.777008 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:24.777744 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:24.777225 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:24.777744 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:24.777329 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:24.985195 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.985163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" event={"ID":"059a60c6-94e4-47a3-b18e-5c347039a27a","Type":"ContainerStarted","Data":"44f419be098917c94879b99c6c182e6b68bde88dcd8bd549773776bf1e4ed962"} Jul 27 23:16:24.986917 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.986893 2564 generic.go:356] "Generic (PLEG): container finished" podID="d92fda41-1b32-4f06-97c5-0d509c87bb90" containerID="90587f892c194b23c0be42f4ed553089ec5088113fd8678312db2b8b1c037867" exitCode=0 Jul 27 23:16:24.987036 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.986966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerDied","Data":"90587f892c194b23c0be42f4ed553089ec5088113fd8678312db2b8b1c037867"} Jul 27 23:16:24.990791 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.990752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" event={"ID":"b9f9724e-dafa-467a-aafe-61b2a5666e86","Type":"ContainerStarted","Data":"9faf5534c4756495177e131ad3f6f0ffd07f0e8c6ec661cad44d51231a51dcea"} Jul 27 23:16:24.991071 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.991049 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:16:24.991138 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:24.991111 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:16:25.000345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:25.000295 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5lpnv" podStartSLOduration=2.800209392 podStartE2EDuration="27.00027721s" totalImagesPullingTime="24.200067818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:24.999930289 +0000 UTC m=+27.797894915" watchObservedRunningTime="2026-07-27 23:16:25.00027721 +0000 UTC m=+27.798241834" Jul 27 23:16:25.006962 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:25.006943 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:16:25.007102 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:25.007089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:16:25.043221 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:25.043156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" podStartSLOduration=6.730384569 podStartE2EDuration="27.043143704s" totalImagesPullingTime="20.312759135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:25.042447942 +0000 UTC m=+27.840412566" watchObservedRunningTime="2026-07-27 23:16:25.043143704 +0000 UTC m=+27.841108306" Jul 27 23:16:25.776811 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:25.776780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:25.777037 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:25.776932 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:25.994652 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:25.993978 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 23:16:26.328440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.328119 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lmnn5"] Jul 27 23:16:26.328558 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.328492 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:26.328611 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:26.328583 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:26.330739 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.330716 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6jq8"] Jul 27 23:16:26.330888 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.330808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:26.330967 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:26.330906 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:26.339017 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.338995 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vw4gr"] Jul 27 23:16:26.339128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.339081 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:26.339192 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:26.339163 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:26.997384 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.997345 2564 generic.go:356] "Generic (PLEG): container finished" podID="d92fda41-1b32-4f06-97c5-0d509c87bb90" containerID="7278deed5a8938e5d6b444742a6ff31fc61e300bd91dae579ebcfbe078a4e972" exitCode=0 Jul 27 23:16:26.997754 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.997440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerDied","Data":"7278deed5a8938e5d6b444742a6ff31fc61e300bd91dae579ebcfbe078a4e972"} Jul 27 23:16:26.997754 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:26.997581 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 27 23:16:27.777002 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:27.776972 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:27.777166 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:27.777063 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:27.777166 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:27.777101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:27.777166 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:27.777141 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:27.777166 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:27.777164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:27.777309 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:27.777208 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:28.020018 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:28.019990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:16:29.003435 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:29.003403 2564 generic.go:356] "Generic (PLEG): container finished" podID="d92fda41-1b32-4f06-97c5-0d509c87bb90" containerID="2f2c751c6ce6e1a76e20abfc315e2ab9e11984fb40583117e1105153fb7b416c" exitCode=0 Jul 27 23:16:29.003580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:29.003479 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerDied","Data":"2f2c751c6ce6e1a76e20abfc315e2ab9e11984fb40583117e1105153fb7b416c"} Jul 27 23:16:29.780342 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:29.780313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:29.780342 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:29.780328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:29.780894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:29.780326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:29.780894 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:29.780417 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:29.780894 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:29.780505 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:29.780894 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:29.780575 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:30.363796 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:30.363764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:30.363978 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:30.363856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:30.363978 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:30.363881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:30.363978 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.363923 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:30.363978 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.363978 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:30.364190 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.364003 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:02.363983728 +0000 UTC m=+65.161948332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 23:16:30.364190 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.364009 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 23:16:30.364190 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.364026 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 23:16:30.364190 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.364032 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret podName:65b91810-92da-4848-b465-2025a4ce7ec0 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:02.364016263 +0000 UTC m=+65.161980865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret") pod "global-pull-secret-syncer-vw4gr" (UID: "65b91810-92da-4848-b465-2025a4ce7ec0") : object "kube-system"/"original-pull-secret" not registered Jul 27 23:16:30.364190 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.364039 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wkdgg for pod openshift-network-diagnostics/network-check-target-lmnn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:30.364190 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:30.364081 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg podName:b2e43f12-7806-4075-b6b8-74a06b44212b nodeName:}" failed. No retries permitted until 2026-07-27 23:17:02.364069295 +0000 UTC m=+65.162033903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkdgg" (UniqueName: "kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg") pod "network-check-target-lmnn5" (UID: "b2e43f12-7806-4075-b6b8-74a06b44212b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 23:16:31.780260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:31.780227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:31.780260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:31.780260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:31.780749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:31.780239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:31.780749 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:31.780347 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vw4gr" podUID="65b91810-92da-4848-b465-2025a4ce7ec0" Jul 27 23:16:31.780749 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:31.780459 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:16:31.780749 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:31.780544 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lmnn5" podUID="b2e43f12-7806-4075-b6b8-74a06b44212b" Jul 27 23:16:31.991093 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:31.991007 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-248.ec2.internal" event="NodeReady" Jul 27 23:16:31.991251 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:31.991156 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 23:16:32.022702 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.022671 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cd8dc995c-frmhn"] Jul 27 23:16:32.044505 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.044478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cd8dc995c-frmhn"] Jul 27 23:16:32.044659 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.044521 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kh2xw"] Jul 27 23:16:32.044659 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.044599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.047174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.046911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 23:16:32.047174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.046922 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 23:16:32.047174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.046975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 23:16:32.047174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.047025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-66lfc\"" Jul 27 23:16:32.053717 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.053694 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 23:16:32.058990 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.058176 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rlfmr"] Jul 27 23:16:32.058990 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.058496 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.061074 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.061008 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 23:16:32.061231 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.061217 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 23:16:32.061371 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.061359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-txnj2\"" Jul 27 23:16:32.061608 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.061592 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 23:16:32.076474 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.076454 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kh2xw"] Jul 27 23:16:32.076571 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.076487 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlfmr"] Jul 27 23:16:32.076617 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.076577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.078784 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.078764 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 23:16:32.078942 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.078925 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 23:16:32.079046 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.079011 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qdw\"" Jul 27 23:16:32.177478 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-trusted-ca\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.177648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.177648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.177648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-installation-pull-secrets\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.177648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gxp\" (UniqueName: \"kubernetes.io/projected/e3deae1b-7472-44a8-881d-26b065db3335-kube-api-access-z6gxp\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.177648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-bound-sa-token\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.177871 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75cddcee-0023-4c16-828c-2294054f6c57-cert\") pod \"ingress-canary-kh2xw\" (UID: \"75cddcee-0023-4c16-828c-2294054f6c57\") " pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.177871 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksgm5\" (UniqueName: \"kubernetes.io/projected/75cddcee-0023-4c16-828c-2294054f6c57-kube-api-access-ksgm5\") pod \"ingress-canary-kh2xw\" (UID: \"75cddcee-0023-4c16-828c-2294054f6c57\") " pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.177871 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3deae1b-7472-44a8-881d-26b065db3335-tmp-dir\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.177871 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-image-registry-private-configuration\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.177871 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-certificates\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.177871 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjl8\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-kube-api-access-ppjl8\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.178078 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3deae1b-7472-44a8-881d-26b065db3335-config-volume\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.178078 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.177937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/560af22b-0e13-4e3b-b21f-c02c8f6011f6-ca-trust-extracted\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.178231 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.178200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:32.258793 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.258744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:32.278587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.278587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-installation-pull-secrets\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gxp\" (UniqueName: \"kubernetes.io/projected/e3deae1b-7472-44a8-881d-26b065db3335-kube-api-access-z6gxp\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-bound-sa-token\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75cddcee-0023-4c16-828c-2294054f6c57-cert\") pod \"ingress-canary-kh2xw\" (UID: \"75cddcee-0023-4c16-828c-2294054f6c57\") " pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.278725 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.278745 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:16:32.278760 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.278763 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.278789 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:32.778768509 +0000 UTC m=+35.576733130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.278846 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:32.778808749 +0000 UTC m=+35.576773357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksgm5\" (UniqueName: \"kubernetes.io/projected/75cddcee-0023-4c16-828c-2294054f6c57-kube-api-access-ksgm5\") pod \"ingress-canary-kh2xw\" (UID: \"75cddcee-0023-4c16-828c-2294054f6c57\") " pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3deae1b-7472-44a8-881d-26b065db3335-tmp-dir\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-image-registry-private-configuration\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-certificates\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.278968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjl8\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-kube-api-access-ppjl8\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.279015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3deae1b-7472-44a8-881d-26b065db3335-config-volume\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.279048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/560af22b-0e13-4e3b-b21f-c02c8f6011f6-ca-trust-extracted\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.279208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.279107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-trusted-ca\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.280318 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.279838 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-certificates\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.280318 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.280129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-trusted-ca\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.280318 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.280201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3deae1b-7472-44a8-881d-26b065db3335-tmp-dir\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.280318 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.280284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/560af22b-0e13-4e3b-b21f-c02c8f6011f6-ca-trust-extracted\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.280543 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.280421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3deae1b-7472-44a8-881d-26b065db3335-config-volume\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.284470 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.284452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75cddcee-0023-4c16-828c-2294054f6c57-cert\") pod \"ingress-canary-kh2xw\" (UID: \"75cddcee-0023-4c16-828c-2294054f6c57\") " pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.284580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.284453 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-installation-pull-secrets\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.287371 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.287350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gxp\" (UniqueName: \"kubernetes.io/projected/e3deae1b-7472-44a8-881d-26b065db3335-kube-api-access-z6gxp\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.287835 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.287793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjl8\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-kube-api-access-ppjl8\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.288009 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.287991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-bound-sa-token\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.288183 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.288163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-image-registry-private-configuration\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.294924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.294904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksgm5\" (UniqueName: \"kubernetes.io/projected/75cddcee-0023-4c16-828c-2294054f6c57-kube-api-access-ksgm5\") pod \"ingress-canary-kh2xw\" (UID: \"75cddcee-0023-4c16-828c-2294054f6c57\") " pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.371285 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.371259 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kh2xw" Jul 27 23:16:32.783311 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.783260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:32.783311 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:32.783311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:32.783871 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.783421 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:16:32.783871 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.783489 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:33.78347115 +0000 UTC m=+36.581435751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:16:32.783871 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.783517 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:16:32.783871 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.783533 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:16:32.783871 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:32.783599 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:33.783568686 +0000 UTC m=+36.581533290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:16:33.352750 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.352722 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kh2xw"] Jul 27 23:16:33.422013 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:16:33.421980 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cddcee_0023_4c16_828c_2294054f6c57.slice/crio-70cc28f7c353d7b4e5dd9459e32b8e055485e959ca19e1cd4f018eaa7b5e90bf WatchSource:0}: Error finding container 70cc28f7c353d7b4e5dd9459e32b8e055485e959ca19e1cd4f018eaa7b5e90bf: Status 404 returned error can't find the container with id 70cc28f7c353d7b4e5dd9459e32b8e055485e959ca19e1cd4f018eaa7b5e90bf Jul 27 23:16:33.776774 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.776746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:16:33.776948 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.776790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:16:33.776948 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.776756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:16:33.780300 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.780279 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 23:16:33.780436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.780317 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 23:16:33.780436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.780320 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 23:16:33.780436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.780341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 23:16:33.780436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.780335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ft576\"" Jul 27 23:16:33.780436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.780284 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n98qx\"" Jul 27 23:16:33.791470 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.791449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:33.792049 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:33.791598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:33.792049 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:33.791616 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:16:33.792049 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:33.791631 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:16:33.792049 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:33.791682 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:35.791665678 +0000 UTC m=+38.589630283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:16:33.792049 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:33.791711 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:16:33.792049 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:33.791768 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:35.791751068 +0000 UTC m=+38.589715671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:16:34.019612 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:34.019581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kh2xw" event={"ID":"75cddcee-0023-4c16-828c-2294054f6c57","Type":"ContainerStarted","Data":"70cc28f7c353d7b4e5dd9459e32b8e055485e959ca19e1cd4f018eaa7b5e90bf"} Jul 27 23:16:34.022329 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:34.022302 2564 generic.go:356] "Generic (PLEG): container finished" podID="d92fda41-1b32-4f06-97c5-0d509c87bb90" containerID="77c96b4c3610628076cb53325fed962132d335594cdf5d31bb8a783388aeba8d" exitCode=0 Jul 27 23:16:34.022456 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:34.022353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerDied","Data":"77c96b4c3610628076cb53325fed962132d335594cdf5d31bb8a783388aeba8d"} Jul 27 23:16:35.029206 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:35.029086 2564 generic.go:356] "Generic (PLEG): container finished" podID="d92fda41-1b32-4f06-97c5-0d509c87bb90" containerID="e44a9dc51a9b6d2214126e72e3c9fbe07bc78dc74638c7aa27490215fd7dd724" exitCode=0 Jul 27 23:16:35.029206 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:35.029149 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerDied","Data":"e44a9dc51a9b6d2214126e72e3c9fbe07bc78dc74638c7aa27490215fd7dd724"} Jul 27 23:16:35.808696 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:35.808504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:35.808889 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:35.808713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:35.808889 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:35.808649 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:16:35.808889 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:35.808839 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:16:35.808889 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:35.808854 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:16:35.808889 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:35.808866 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:39.808845399 +0000 UTC m=+42.606810002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:16:35.808889 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:35.808887 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:39.808877114 +0000 UTC m=+42.606841735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:16:36.035508 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:36.035470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" event={"ID":"d92fda41-1b32-4f06-97c5-0d509c87bb90","Type":"ContainerStarted","Data":"b7ef15e251839778f3f9fb7b3dee3953eb645fc9207280de49c576f43f48b784"} Jul 27 23:16:36.058215 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:36.058165 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gd2zk" podStartSLOduration=6.91563519 podStartE2EDuration="38.058143247s" totalImagesPullingTime="31.142508057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:36.056356327 +0000 UTC m=+38.854320952" watchObservedRunningTime="2026-07-27 23:16:36.058143247 +0000 UTC m=+38.856107875" Jul 27 23:16:36.180380 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:36.180300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:36.236552 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:36.236516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:37.203012 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:37.202976 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:38.041083 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:38.041049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kh2xw" event={"ID":"75cddcee-0023-4c16-828c-2294054f6c57","Type":"ContainerStarted","Data":"2a8b177a5054abf3a3d67257d785876d1fecff2a6ed5a19123e7ef357ae786c1"} Jul 27 23:16:38.055229 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:38.055182 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kh2xw" podStartSLOduration=2.475049507 podStartE2EDuration="6.055165466s" totalImagesPullingTime="3.580115959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:16:38.054505742 +0000 UTC m=+40.852470383" watchObservedRunningTime="2026-07-27 23:16:38.055165466 +0000 UTC m=+40.853130095" Jul 27 23:16:38.815579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:38.815541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:38.876243 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:38.876206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:38.974165 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:38.974138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:39.293262 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:39.293230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:39.343923 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:39.343893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:39.828856 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:39.828809 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:39.844752 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:39.844725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:39.844867 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:39.844755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:39.844917 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:39.844875 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:16:39.844917 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:39.844900 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:16:39.844917 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:39.844911 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:16:39.845026 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:39.844950 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:47.844933357 +0000 UTC m=+50.642897963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:16:39.845026 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:39.844964 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:16:47.844957709 +0000 UTC m=+50.642922311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:16:39.926498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:39.926471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:40.905274 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:40.905227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:40.951730 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:40.951692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:41.888300 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:41.888264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:41.936740 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:41.936709 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:42.833574 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:42.833536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:47.073147 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:47.073115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:47.901690 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:47.901638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:16:47.901690 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:47.901683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:16:47.901959 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:47.901795 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:16:47.901959 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:47.901798 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:16:47.901959 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:47.901882 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:03.901866935 +0000 UTC m=+66.699831536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:16:47.901959 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:47.901883 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:16:47.901959 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:16:47.901933 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:03.901917749 +0000 UTC m=+66.699882356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:16:51.554012 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:51.553978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:52.557672 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:52.557636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:52.608345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:52.608315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:54.900102 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:54.900054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:55.000810 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:55.000771 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:55.100594 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:55.100563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:55.148738 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:55.148705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:57.793163 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:57.793135 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:57.842088 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:57.842061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:58.031125 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:58.031101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67nrk" Jul 27 23:16:58.180283 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:58.180207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:16:58.231945 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:16:58.231907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:17:02.401109 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.401070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:17:02.401109 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.401108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:17:02.401556 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.401148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:17:02.403654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.403638 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 23:17:02.403704 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.403679 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 23:17:02.403751 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.403677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 23:17:02.412337 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:02.412316 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 23:17:02.412439 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:02.412392 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:18:06.412372262 +0000 UTC m=+129.210336882 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : secret "metrics-daemon-secret" not found Jul 27 23:17:02.414099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.414083 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 23:17:02.415701 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.415683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65b91810-92da-4848-b465-2025a4ce7ec0-original-pull-secret\") pod \"global-pull-secret-syncer-vw4gr\" (UID: \"65b91810-92da-4848-b465-2025a4ce7ec0\") " pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:17:02.424997 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.424976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdgg\" (UniqueName: \"kubernetes.io/projected/b2e43f12-7806-4075-b6b8-74a06b44212b-kube-api-access-wkdgg\") pod \"network-check-target-lmnn5\" (UID: \"b2e43f12-7806-4075-b6b8-74a06b44212b\") " pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:17:02.588908 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.588883 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ft576\"" Jul 27 23:17:02.591451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.591439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vw4gr" Jul 27 23:17:02.597309 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.597293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:17:02.734455 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.734429 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vw4gr"] Jul 27 23:17:02.737930 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:17:02.737906 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b91810_92da_4848_b465_2025a4ce7ec0.slice/crio-22adf1dc91d29c65da8783fc33e02535a8030fd28f123fea0829c81785cb89d0 WatchSource:0}: Error finding container 22adf1dc91d29c65da8783fc33e02535a8030fd28f123fea0829c81785cb89d0: Status 404 returned error can't find the container with id 22adf1dc91d29c65da8783fc33e02535a8030fd28f123fea0829c81785cb89d0 Jul 27 23:17:02.750281 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:02.750257 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lmnn5"] Jul 27 23:17:02.753246 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:17:02.753222 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e43f12_7806_4075_b6b8_74a06b44212b.slice/crio-dfe104d881c939f7ab98ad8b9052b5667689a15ff18bd62f0b6bf2afc7628f27 WatchSource:0}: Error finding container dfe104d881c939f7ab98ad8b9052b5667689a15ff18bd62f0b6bf2afc7628f27: Status 404 returned error can't find the container with id dfe104d881c939f7ab98ad8b9052b5667689a15ff18bd62f0b6bf2afc7628f27 Jul 27 23:17:03.099773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:03.099731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vw4gr" event={"ID":"65b91810-92da-4848-b465-2025a4ce7ec0","Type":"ContainerStarted","Data":"22adf1dc91d29c65da8783fc33e02535a8030fd28f123fea0829c81785cb89d0"} Jul 27 23:17:03.100731 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:03.100709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lmnn5" event={"ID":"b2e43f12-7806-4075-b6b8-74a06b44212b","Type":"ContainerStarted","Data":"dfe104d881c939f7ab98ad8b9052b5667689a15ff18bd62f0b6bf2afc7628f27"} Jul 27 23:17:03.911737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:03.911686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:17:03.912223 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:03.911747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:17:03.912223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:03.911882 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:17:03.912223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:03.911961 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:35.911940702 +0000 UTC m=+98.709905309 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:17:03.912223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:03.911997 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:17:03.912223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:03.912019 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:17:03.912223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:03.912080 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:17:35.912063635 +0000 UTC m=+98.710028251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:17:08.114638 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:08.114546 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lmnn5" event={"ID":"b2e43f12-7806-4075-b6b8-74a06b44212b","Type":"ContainerStarted","Data":"b71ecd26b59dda0dbe4e15cc96510d89bbd54a89a426160e753cb056002d4ca9"} Jul 27 23:17:08.115092 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:08.114650 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:17:08.116167 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:08.116144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vw4gr" event={"ID":"65b91810-92da-4848-b465-2025a4ce7ec0","Type":"ContainerStarted","Data":"91189c8803a4b613dbce948ccf9053592e9d03a2b88f8fb47851777d7ec1848a"} Jul 27 23:17:08.134191 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:08.134150 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lmnn5" podStartSLOduration=65.120208966 podStartE2EDuration="1m10.134139286s" totalImagesPullingTime="5.01393032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:08.133905205 +0000 UTC m=+70.931869831" watchObservedRunningTime="2026-07-27 23:17:08.134139286 +0000 UTC m=+70.932103901" Jul 27 23:17:15.879081 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:15.879047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:17:15.927504 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:15.927465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:17:17.911123 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:17.911091 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:17:17.961841 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:17.961802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:17:35.938892 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:35.938845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:17:35.938892 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:35.938897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:17:35.939223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:35.939010 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:17:35.939223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:35.939009 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:17:35.939223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:35.939076 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:18:39.93906022 +0000 UTC m=+162.737024823 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:17:35.939223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:35.939023 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:17:35.939223 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:17:35.939153 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:18:39.939141713 +0000 UTC m=+162.737106316 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:17:39.120928 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:39.120900 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lmnn5" Jul 27 23:17:39.135667 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:17:39.135623 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vw4gr" podStartSLOduration=96.094643668 podStartE2EDuration="1m41.135609735s" totalImagesPullingTime="5.040966067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:17:08.149467368 +0000 UTC m=+70.947431992" watchObservedRunningTime="2026-07-27 23:17:39.135609735 +0000 UTC m=+101.933574366" Jul 27 23:18:06.434046 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:18:06.434005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:18:06.434525 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:06.434119 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 23:18:06.434525 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:06.434172 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:08.434158015 +0000 UTC m=+251.232122618 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : secret "metrics-daemon-secret" not found Jul 27 23:18:35.056430 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:35.056384 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" podUID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" Jul 27 23:18:35.086658 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:35.086638 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rlfmr" podUID="e3deae1b-7472-44a8-881d-26b065db3335" Jul 27 23:18:36.796893 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:36.796840 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:18:39.954651 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:18:39.954615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:18:39.954651 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:18:39.954648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:18:39.955080 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:39.954751 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 23:18:39.955080 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:39.954750 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 23:18:39.955080 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:39.954762 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cd8dc995c-frmhn: secret "image-registry-tls" not found Jul 27 23:18:39.955080 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:39.954842 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls podName:e3deae1b-7472-44a8-881d-26b065db3335 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:41.95480021 +0000 UTC m=+284.752764812 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls") pod "dns-default-rlfmr" (UID: "e3deae1b-7472-44a8-881d-26b065db3335") : secret "dns-default-metrics-tls" not found Jul 27 23:18:39.955080 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:18:39.954861 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls podName:560af22b-0e13-4e3b-b21f-c02c8f6011f6 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:41.954854081 +0000 UTC m=+284.752818683 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls") pod "image-registry-6cd8dc995c-frmhn" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6") : secret "image-registry-tls" not found Jul 27 23:18:49.776485 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:18:49.776407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:18:49.776895 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:18:49.776598 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlfmr" Jul 27 23:18:50.776772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:18:50.776734 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:19:32.211161 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:32.211128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:54.703512 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.703479 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9"] Jul 27 23:19:54.705545 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.705528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.709239 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.709211 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm"] Jul 27 23:19:54.709443 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.709423 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zp7ch\"" Jul 27 23:19:54.709520 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.709477 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 23:19:54.709618 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.709517 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:19:54.709709 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.709691 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 23:19:54.709770 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.709725 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 23:19:54.711021 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.711002 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl"] Jul 27 23:19:54.711139 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.711116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.712992 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.712974 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" Jul 27 23:19:54.713552 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.713538 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 23:19:54.713934 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.713917 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h5f97\"" Jul 27 23:19:54.714102 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.714088 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 23:19:54.714159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.714088 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 23:19:54.716852 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.716835 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-86bxk\"" Jul 27 23:19:54.719349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.719329 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 23:19:54.720988 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.720970 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9"] Jul 27 23:19:54.724423 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.724391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl"] Jul 27 23:19:54.727384 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.727360 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm"] Jul 27 23:19:54.801138 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.801098 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5q72l"] Jul 27 23:19:54.803348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.803323 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:54.804731 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.804705 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh"] Jul 27 23:19:54.806018 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.806001 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5fdqh\"" Jul 27 23:19:54.806273 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.806260 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 23:19:54.806323 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.806268 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 23:19:54.806862 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.806848 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7mjm7"] Jul 27 23:19:54.806961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.806949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" Jul 27 23:19:54.808756 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.808737 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74f9f647d5-fbgwn"] Jul 27 23:19:54.808852 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.808845 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.808935 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.808922 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-n7m2g\"" Jul 27 23:19:54.809265 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.809245 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:19:54.809348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.809277 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 23:19:54.810558 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.810538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:54.811220 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.810999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 23:19:54.811332 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.811235 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 23:19:54.811332 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.811288 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 23:19:54.811652 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.811631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gdf2x\"" Jul 27 23:19:54.811747 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.811686 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 23:19:54.812279 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.812262 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 23:19:54.812615 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.812602 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vt5xb\"" Jul 27 23:19:54.812810 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.812795 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 23:19:54.812938 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.812919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 23:19:54.813039 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.812997 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 23:19:54.813039 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.813001 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 23:19:54.813310 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.813294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 23:19:54.814749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.814729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5q72l"] Jul 27 23:19:54.818042 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.818023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh"] Jul 27 23:19:54.819242 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.819224 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7mjm7"] Jul 27 23:19:54.819624 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.819604 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 23:19:54.833760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.833736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74f9f647d5-fbgwn"] Jul 27 23:19:54.848635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848609 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/954778d1-cf9f-46d3-87d3-4e75c6c292ed-config\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.848855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3fa7c20-89da-424a-9c08-7053736e0cf6-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.848855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcv2d\" (UniqueName: \"kubernetes.io/projected/39b716b7-e66b-43aa-b0ff-532f50268c68-kube-api-access-kcv2d\") pod \"network-check-source-7d88cb55d8-7nmvl\" (UID: \"39b716b7-e66b-43aa-b0ff-532f50268c68\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" Jul 27 23:19:54.848855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmlr\" (UniqueName: \"kubernetes.io/projected/954778d1-cf9f-46d3-87d3-4e75c6c292ed-kube-api-access-wqmlr\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.848855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbzc\" (UniqueName: \"kubernetes.io/projected/e3fa7c20-89da-424a-9c08-7053736e0cf6-kube-api-access-wdbzc\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.848855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/954778d1-cf9f-46d3-87d3-4e75c6c292ed-serving-cert\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.849095 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.848867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.945037 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.944997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:54.950126 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmlr\" (UniqueName: \"kubernetes.io/projected/954778d1-cf9f-46d3-87d3-4e75c6c292ed-kube-api-access-wqmlr\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.950218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbzc\" (UniqueName: \"kubernetes.io/projected/e3fa7c20-89da-424a-9c08-7053736e0cf6-kube-api-access-wdbzc\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.950218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-default-certificate\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:54.950218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.950330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c23353e-81b9-4556-a3a2-33b243c63f68-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.950330 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:54.950265 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:54.950330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:54.950330 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:54.950327 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls podName:e3fa7c20-89da-424a-9c08-7053736e0cf6 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:55.450308832 +0000 UTC m=+238.248273434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slmlm" (UID: "e3fa7c20-89da-424a-9c08-7053736e0cf6") : secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:54.950481 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c23353e-81b9-4556-a3a2-33b243c63f68-service-ca-bundle\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.950481 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/954778d1-cf9f-46d3-87d3-4e75c6c292ed-config\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.950481 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:54.950481 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcv2d\" (UniqueName: \"kubernetes.io/projected/39b716b7-e66b-43aa-b0ff-532f50268c68-kube-api-access-kcv2d\") pod \"network-check-source-7d88cb55d8-7nmvl\" (UID: \"39b716b7-e66b-43aa-b0ff-532f50268c68\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" Jul 27 23:19:54.950614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c23353e-81b9-4556-a3a2-33b243c63f68-snapshots\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.950614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c23353e-81b9-4556-a3a2-33b243c63f68-serving-cert\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.950614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/954778d1-cf9f-46d3-87d3-4e75c6c292ed-serving-cert\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.950714 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c23353e-81b9-4556-a3a2-33b243c63f68-tmp\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.950714 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:54.950714 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dcf565a0-b02a-4958-9786-02ce54f8c298-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:54.950833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950722 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3fa7c20-89da-424a-9c08-7053736e0cf6-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.950833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhws\" (UniqueName: \"kubernetes.io/projected/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-kube-api-access-gnhws\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:54.950833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqgm\" (UniqueName: \"kubernetes.io/projected/12e64263-e8d2-4959-b3c4-3ad2fc7c2c20-kube-api-access-psqgm\") pod \"volume-data-source-validator-549c6d54cf-zp4sh\" (UID: \"12e64263-e8d2-4959-b3c4-3ad2fc7c2c20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" Jul 27 23:19:54.950941 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9fb\" (UniqueName: \"kubernetes.io/projected/9c23353e-81b9-4556-a3a2-33b243c63f68-kube-api-access-gb9fb\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:54.950941 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.950880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-stats-auth\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:54.951410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.951393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3fa7c20-89da-424a-9c08-7053736e0cf6-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.951456 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.951428 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/954778d1-cf9f-46d3-87d3-4e75c6c292ed-config\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.953329 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.953310 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/954778d1-cf9f-46d3-87d3-4e75c6c292ed-serving-cert\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.958173 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.958104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmlr\" (UniqueName: \"kubernetes.io/projected/954778d1-cf9f-46d3-87d3-4e75c6c292ed-kube-api-access-wqmlr\") pod \"service-ca-operator-5d6c75f84f-kkmc9\" (UID: \"954778d1-cf9f-46d3-87d3-4e75c6c292ed\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:54.958271 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.958196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbzc\" (UniqueName: \"kubernetes.io/projected/e3fa7c20-89da-424a-9c08-7053736e0cf6-kube-api-access-wdbzc\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:54.958336 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.958305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcv2d\" (UniqueName: \"kubernetes.io/projected/39b716b7-e66b-43aa-b0ff-532f50268c68-kube-api-access-kcv2d\") pod \"network-check-source-7d88cb55d8-7nmvl\" (UID: \"39b716b7-e66b-43aa-b0ff-532f50268c68\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" Jul 27 23:19:54.993205 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:54.993135 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:55.015883 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.015851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" Jul 27 23:19:55.030461 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.030429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" Jul 27 23:19:55.051291 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c23353e-81b9-4556-a3a2-33b243c63f68-tmp\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.051291 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.051490 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dcf565a0-b02a-4958-9786-02ce54f8c298-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:55.051490 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.051431 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 23:19:55.051552 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhws\" (UniqueName: \"kubernetes.io/projected/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-kube-api-access-gnhws\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.051552 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.051501 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:55.551481667 +0000 UTC m=+238.349446290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : secret "router-metrics-certs-default" not found Jul 27 23:19:55.051552 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqgm\" (UniqueName: \"kubernetes.io/projected/12e64263-e8d2-4959-b3c4-3ad2fc7c2c20-kube-api-access-psqgm\") pod \"volume-data-source-validator-549c6d54cf-zp4sh\" (UID: \"12e64263-e8d2-4959-b3c4-3ad2fc7c2c20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" Jul 27 23:19:55.051660 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9fb\" (UniqueName: \"kubernetes.io/projected/9c23353e-81b9-4556-a3a2-33b243c63f68-kube-api-access-gb9fb\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.051660 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-stats-auth\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.051660 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-default-certificate\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.051749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c23353e-81b9-4556-a3a2-33b243c63f68-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.051749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c23353e-81b9-4556-a3a2-33b243c63f68-tmp\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.051749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.051749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c23353e-81b9-4556-a3a2-33b243c63f68-service-ca-bundle\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.051910 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:55.051910 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c23353e-81b9-4556-a3a2-33b243c63f68-snapshots\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.051910 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.051863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c23353e-81b9-4556-a3a2-33b243c63f68-serving-cert\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.052769 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.052080 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dcf565a0-b02a-4958-9786-02ce54f8c298-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:55.052769 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.052171 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 23:19:55.052769 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.052178 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:55.552159269 +0000 UTC m=+238.350123875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : configmap references non-existent config key: service-ca.crt Jul 27 23:19:55.052769 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.052208 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert podName:dcf565a0-b02a-4958-9786-02ce54f8c298 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:55.552197308 +0000 UTC m=+238.350161916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5q72l" (UID: "dcf565a0-b02a-4958-9786-02ce54f8c298") : secret "networking-console-plugin-cert" not found Jul 27 23:19:55.052769 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.052712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c23353e-81b9-4556-a3a2-33b243c63f68-service-ca-bundle\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.053130 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.053014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c23353e-81b9-4556-a3a2-33b243c63f68-snapshots\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.053130 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.053061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c23353e-81b9-4556-a3a2-33b243c63f68-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.054362 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.054339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-stats-auth\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.054982 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.054964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c23353e-81b9-4556-a3a2-33b243c63f68-serving-cert\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.055058 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.055041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-default-certificate\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.060320 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.060285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqgm\" (UniqueName: \"kubernetes.io/projected/12e64263-e8d2-4959-b3c4-3ad2fc7c2c20-kube-api-access-psqgm\") pod \"volume-data-source-validator-549c6d54cf-zp4sh\" (UID: \"12e64263-e8d2-4959-b3c4-3ad2fc7c2c20\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" Jul 27 23:19:55.060783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.060764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9fb\" (UniqueName: \"kubernetes.io/projected/9c23353e-81b9-4556-a3a2-33b243c63f68-kube-api-access-gb9fb\") pod \"insights-operator-5885fdb759-7mjm7\" (UID: \"9c23353e-81b9-4556-a3a2-33b243c63f68\") " pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.060900 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.060865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhws\" (UniqueName: \"kubernetes.io/projected/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-kube-api-access-gnhws\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.124125 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.124096 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" Jul 27 23:19:55.131245 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.131020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" Jul 27 23:19:55.151313 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.151282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9"] Jul 27 23:19:55.172205 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.172175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl"] Jul 27 23:19:55.176053 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:19:55.176023 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b716b7_e66b_43aa_b0ff_532f50268c68.slice/crio-b7dbf92d805207ab61cc412e7a37c800b94429764cf445a8ac9c84d76e221654 WatchSource:0}: Error finding container b7dbf92d805207ab61cc412e7a37c800b94429764cf445a8ac9c84d76e221654: Status 404 returned error can't find the container with id b7dbf92d805207ab61cc412e7a37c800b94429764cf445a8ac9c84d76e221654 Jul 27 23:19:55.268029 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.267951 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh"] Jul 27 23:19:55.271634 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:19:55.271576 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e64263_e8d2_4959_b3c4_3ad2fc7c2c20.slice/crio-5db989a828c02af2269919f04821d9f080c7240df137f4d862aa7e1e0942c37b WatchSource:0}: Error finding container 5db989a828c02af2269919f04821d9f080c7240df137f4d862aa7e1e0942c37b: Status 404 returned error can't find the container with id 5db989a828c02af2269919f04821d9f080c7240df137f4d862aa7e1e0942c37b Jul 27 23:19:55.287729 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.287691 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-7mjm7"] Jul 27 23:19:55.291323 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:19:55.291293 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c23353e_81b9_4556_a3a2_33b243c63f68.slice/crio-f44a2a2dd46636da9ab8b5322ef30b8b70c601362e168dc458ad0f3d1e41ac0c WatchSource:0}: Error finding container f44a2a2dd46636da9ab8b5322ef30b8b70c601362e168dc458ad0f3d1e41ac0c: Status 404 returned error can't find the container with id f44a2a2dd46636da9ab8b5322ef30b8b70c601362e168dc458ad0f3d1e41ac0c Jul 27 23:19:55.455143 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.455098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:55.455319 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.455235 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:55.455319 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.455301 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls podName:e3fa7c20-89da-424a-9c08-7053736e0cf6 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:56.455286009 +0000 UTC m=+239.253250615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slmlm" (UID: "e3fa7c20-89da-424a-9c08-7053736e0cf6") : secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:55.502909 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.502874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" event={"ID":"9c23353e-81b9-4556-a3a2-33b243c63f68","Type":"ContainerStarted","Data":"f44a2a2dd46636da9ab8b5322ef30b8b70c601362e168dc458ad0f3d1e41ac0c"} Jul 27 23:19:55.503925 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.503904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" event={"ID":"12e64263-e8d2-4959-b3c4-3ad2fc7c2c20","Type":"ContainerStarted","Data":"5db989a828c02af2269919f04821d9f080c7240df137f4d862aa7e1e0942c37b"} Jul 27 23:19:55.504949 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.504926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" event={"ID":"954778d1-cf9f-46d3-87d3-4e75c6c292ed","Type":"ContainerStarted","Data":"5fa0e331087265c9297981c38c0045af4ec437d5ede4ad49e5f532d493f9a949"} Jul 27 23:19:55.506373 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.506352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" event={"ID":"39b716b7-e66b-43aa-b0ff-532f50268c68","Type":"ContainerStarted","Data":"a183ad4f3481754338987dc4ec0c5d48e3ab3fdf5948628038cedde4c120cf6f"} Jul 27 23:19:55.506450 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.506391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" event={"ID":"39b716b7-e66b-43aa-b0ff-532f50268c68","Type":"ContainerStarted","Data":"b7dbf92d805207ab61cc412e7a37c800b94429764cf445a8ac9c84d76e221654"} Jul 27 23:19:55.520957 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.520905 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7nmvl" podStartSLOduration=1.5208921370000001 podStartE2EDuration="1.520892137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:19:55.520248206 +0000 UTC m=+238.318212831" watchObservedRunningTime="2026-07-27 23:19:55.520892137 +0000 UTC m=+238.318856761" Jul 27 23:19:55.556259 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.556227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.556395 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.556264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:55.556395 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:55.556303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:55.556471 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.556401 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 23:19:55.556471 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.556415 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 23:19:55.556471 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.556447 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:56.556434441 +0000 UTC m=+239.354399045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : secret "router-metrics-certs-default" not found Jul 27 23:19:55.556575 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.556479 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert podName:dcf565a0-b02a-4958-9786-02ce54f8c298 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:56.556462356 +0000 UTC m=+239.354426965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5q72l" (UID: "dcf565a0-b02a-4958-9786-02ce54f8c298") : secret "networking-console-plugin-cert" not found Jul 27 23:19:55.556575 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:55.556514 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:56.556502565 +0000 UTC m=+239.354467170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : configmap references non-existent config key: service-ca.crt Jul 27 23:19:56.465685 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:56.465635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:56.466139 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.465787 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:56.466139 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.465891 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls podName:e3fa7c20-89da-424a-9c08-7053736e0cf6 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:58.465866247 +0000 UTC m=+241.263830872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slmlm" (UID: "e3fa7c20-89da-424a-9c08-7053736e0cf6") : secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:56.566601 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:56.566574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:56.566711 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:56.566629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:56.566711 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:56.566690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:56.566855 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.566739 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:58.566720367 +0000 UTC m=+241.364684986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : configmap references non-existent config key: service-ca.crt Jul 27 23:19:56.566855 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.566791 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 23:19:56.566855 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.566805 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 23:19:56.566991 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.566925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:58.566907089 +0000 UTC m=+241.364871713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : secret "router-metrics-certs-default" not found Jul 27 23:19:56.566991 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:56.566946 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert podName:dcf565a0-b02a-4958-9786-02ce54f8c298 nodeName:}" failed. No retries permitted until 2026-07-27 23:19:58.566937161 +0000 UTC m=+241.364901765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5q72l" (UID: "dcf565a0-b02a-4958-9786-02ce54f8c298") : secret "networking-console-plugin-cert" not found Jul 27 23:19:57.514434 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:57.514393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" event={"ID":"12e64263-e8d2-4959-b3c4-3ad2fc7c2c20","Type":"ContainerStarted","Data":"476d13959258e0c30f6b2e82881a1abea2f96cdf7cd54a03c0901b0c0ad41297"} Jul 27 23:19:57.528805 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:57.528750 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-zp4sh" podStartSLOduration=2.281254596 podStartE2EDuration="3.528736807s" totalImagesPullingTime="1.247482211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:19:57.527474799 +0000 UTC m=+240.325439423" watchObservedRunningTime="2026-07-27 23:19:57.528736807 +0000 UTC m=+240.326701431" Jul 27 23:19:57.599344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:57.599314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:57.700296 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:57.700261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:58.482507 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.482468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:19:58.482684 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.482603 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:58.482684 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.482669 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls podName:e3fa7c20-89da-424a-9c08-7053736e0cf6 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:02.482653231 +0000 UTC m=+245.280617832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slmlm" (UID: "e3fa7c20-89da-424a-9c08-7053736e0cf6") : secret "cluster-monitoring-operator-tls" not found Jul 27 23:19:58.519067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.519036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" event={"ID":"9c23353e-81b9-4556-a3a2-33b243c63f68","Type":"ContainerStarted","Data":"cbb603943eccaed273fdc355cb2c59fb1782470ca822c2f137d7a45dd28021b2"} Jul 27 23:19:58.523290 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.523261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" event={"ID":"954778d1-cf9f-46d3-87d3-4e75c6c292ed","Type":"ContainerStarted","Data":"4b1145bac0ca6d7e5f5415f98367cc793f7973ff61428268c52fe0d9d2dc9441"} Jul 27 23:19:58.536343 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.536294 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" podStartSLOduration=1.9686406079999998 podStartE2EDuration="4.536275388s" totalImagesPullingTime="2.56763478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:19:58.534924036 +0000 UTC m=+241.332888662" watchObservedRunningTime="2026-07-27 23:19:58.536275388 +0000 UTC m=+241.334239992" Jul 27 23:19:58.549260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.549211 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" podStartSLOduration=1.8458974910000001 podStartE2EDuration="4.549193177s" totalImagesPullingTime="2.703295686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:19:58.547990994 +0000 UTC m=+241.345955614" watchObservedRunningTime="2026-07-27 23:19:58.549193177 +0000 UTC m=+241.347157805" Jul 27 23:19:58.583071 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.583042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:58.583184 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.583091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:19:58.583184 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:58.583148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:19:58.583308 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.583187 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:02.583168351 +0000 UTC m=+245.381132968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : configmap references non-existent config key: service-ca.crt Jul 27 23:19:58.583308 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.583250 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 23:19:58.583308 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.583266 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 23:19:58.583308 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.583301 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:02.583285357 +0000 UTC m=+245.381249961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : secret "router-metrics-certs-default" not found Jul 27 23:19:58.583498 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:58.583320 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert podName:dcf565a0-b02a-4958-9786-02ce54f8c298 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:02.583307029 +0000 UTC m=+245.381271655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5q72l" (UID: "dcf565a0-b02a-4958-9786-02ce54f8c298") : secret "networking-console-plugin-cert" not found Jul 27 23:19:59.214107 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.214067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:59.309660 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.309619 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:59.360256 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.360224 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4whwh"] Jul 27 23:19:59.363334 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.363303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.365579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.365553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 23:19:59.365810 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.365784 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26z57\"" Jul 27 23:19:59.365810 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.365635 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 23:19:59.373917 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.373887 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4whwh"] Jul 27 23:19:59.390097 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.390068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:59.443210 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.443185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:59.490374 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.490308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.490374 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.490361 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.490535 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.490426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmlrq\" (UniqueName: \"kubernetes.io/projected/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-api-access-nmlrq\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.490535 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.490462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-data-volume\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.490535 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.490523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-crio-socket\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.591405 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmlrq\" (UniqueName: \"kubernetes.io/projected/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-api-access-nmlrq\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:59.591487 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-data-volume\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:59.591528 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:59.591550 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:00.091536752 +0000 UTC m=+242.889501353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : configmap "kube-rbac-proxy" not found Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:19:59.591601 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:00.091576488 +0000 UTC m=+242.889541092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : secret "insights-runtime-extractor-tls" not found Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-crio-socket\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.591785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-crio-socket\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.592142 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.591861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-data-volume\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.599212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.599191 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmlrq\" (UniqueName: \"kubernetes.io/projected/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-api-access-nmlrq\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:19:59.755931 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.755904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:19:59.807165 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:19:59.807138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:00.062568 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:00.062496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:00.094833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:00.094793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:00.094946 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:00.094894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:00.094946 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:00.094917 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:20:00.095016 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:00.094986 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:01.094968339 +0000 UTC m=+243.892932963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : configmap "kube-rbac-proxy" not found Jul 27 23:20:00.095063 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:00.095035 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:20:00.095102 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:00.095090 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:01.095076668 +0000 UTC m=+243.893041269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : secret "insights-runtime-extractor-tls" not found Jul 27 23:20:00.164027 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:00.163993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:01.104558 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.104512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:01.104965 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.104576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:01.104965 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:01.104647 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:20:01.104965 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:01.104735 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:20:01.104965 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:01.104746 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:03.104722902 +0000 UTC m=+245.902687530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : configmap "kube-rbac-proxy" not found Jul 27 23:20:01.104965 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:01.104777 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:03.10476664 +0000 UTC m=+245.902731244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : secret "insights-runtime-extractor-tls" not found Jul 27 23:20:01.318091 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.318062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:01.376689 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.376607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:01.632757 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.632679 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rfspg"] Jul 27 23:20:01.634702 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.634686 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.636944 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.636923 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 27 23:20:01.637043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.636944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 27 23:20:01.637734 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.637718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 27 23:20:01.637773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.637740 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7f8vl\"" Jul 27 23:20:01.637773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.637718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 27 23:20:01.643899 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.643879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rfspg"] Jul 27 23:20:01.667120 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.667098 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:01.721708 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.721658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:01.810559 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.810531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-signing-key\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.810714 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.810683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-signing-cabundle\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.810792 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.810747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqz4j\" (UniqueName: \"kubernetes.io/projected/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-kube-api-access-wqz4j\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.912112 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.912026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-signing-cabundle\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.912112 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.912086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqz4j\" (UniqueName: \"kubernetes.io/projected/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-kube-api-access-wqz4j\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.912288 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.912132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-signing-key\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.912707 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.912678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-signing-cabundle\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.914628 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.914602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-signing-key\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.920777 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.920744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqz4j\" (UniqueName: \"kubernetes.io/projected/f22b9c55-276d-4425-b3bf-7d66b1aafa9a-kube-api-access-wqz4j\") pod \"service-ca-658d7b7c78-rfspg\" (UID: \"f22b9c55-276d-4425-b3bf-7d66b1aafa9a\") " pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:01.943737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:01.943717 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" Jul 27 23:20:02.063543 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.063504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rfspg"] Jul 27 23:20:02.066434 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:02.066408 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22b9c55_276d_4425_b3bf_7d66b1aafa9a.slice/crio-c862f2fbc718cb40497747e374e5b1a77041a67f74bd09b254e53cec30f47191 WatchSource:0}: Error finding container c862f2fbc718cb40497747e374e5b1a77041a67f74bd09b254e53cec30f47191: Status 404 returned error can't find the container with id c862f2fbc718cb40497747e374e5b1a77041a67f74bd09b254e53cec30f47191 Jul 27 23:20:02.356682 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.356654 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlxxm_103dec8d-2e5e-4973-b634-b28b043cfb6b/dns-node-resolver/0.log" Jul 27 23:20:02.516874 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.516831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:20:02.517037 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.516998 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 23:20:02.517077 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.517068 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls podName:e3fa7c20-89da-424a-9c08-7053736e0cf6 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:10.517051971 +0000 UTC m=+253.315016573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slmlm" (UID: "e3fa7c20-89da-424a-9c08-7053736e0cf6") : secret "cluster-monitoring-operator-tls" not found Jul 27 23:20:02.536784 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.536752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" event={"ID":"f22b9c55-276d-4425-b3bf-7d66b1aafa9a","Type":"ContainerStarted","Data":"924ae8b9cf922c0c5e8151367cf3d2fd2e5ec7b9eb433b594e8fe92bed3c7f74"} Jul 27 23:20:02.536970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.536845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" event={"ID":"f22b9c55-276d-4425-b3bf-7d66b1aafa9a","Type":"ContainerStarted","Data":"c862f2fbc718cb40497747e374e5b1a77041a67f74bd09b254e53cec30f47191"} Jul 27 23:20:02.555990 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.555946 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-rfspg" podStartSLOduration=1.555930954 podStartE2EDuration="1.555930954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:02.555154171 +0000 UTC m=+245.353118796" watchObservedRunningTime="2026-07-27 23:20:02.555930954 +0000 UTC m=+245.353895634" Jul 27 23:20:02.618619 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.618536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:02.618754 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.618619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:20:02.618754 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.618699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:02.618754 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.618714 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:10.618693527 +0000 UTC m=+253.416658151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : configmap references non-existent config key: service-ca.crt Jul 27 23:20:02.618950 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.618778 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 23:20:02.618950 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.618789 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 23:20:02.618950 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.618846 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs podName:26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:10.618832434 +0000 UTC m=+253.416797053 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs") pod "router-default-74f9f647d5-fbgwn" (UID: "26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5") : secret "router-metrics-certs-default" not found Jul 27 23:20:02.618950 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:02.618888 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert podName:dcf565a0-b02a-4958-9786-02ce54f8c298 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:10.618868735 +0000 UTC m=+253.416833356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5q72l" (UID: "dcf565a0-b02a-4958-9786-02ce54f8c298") : secret "networking-console-plugin-cert" not found Jul 27 23:20:02.637979 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.637952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:02.739717 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.739683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:02.943917 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:02.943844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.044849 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.044797 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.124313 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.124277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:03.124583 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:03.124403 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:20:03.124675 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:03.124663 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:07.124621369 +0000 UTC m=+249.922585974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : configmap "kube-rbac-proxy" not found Jul 27 23:20:03.124741 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.124660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:03.124937 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:03.124920 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:20:03.125116 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:03.125086 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:07.125070606 +0000 UTC m=+249.923035210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : secret "insights-runtime-extractor-tls" not found Jul 27 23:20:03.145078 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.145038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.245647 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.245540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.345983 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.345944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.358361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.358329 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wld4r_ff37ceb7-f0de-44f8-82a6-c34d3899c294/node-ca/0.log" Jul 27 23:20:03.502408 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.502378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.639185 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.639152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:03.956685 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:03.956612 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kh2xw_75cddcee-0023-4c16-828c-2294054f6c57/serve-healthcheck-canary/0.log" Jul 27 23:20:04.826674 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:04.826641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:05.029752 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:05.029726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:05.237751 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:05.237660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:05.428248 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:05.428217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:05.626038 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:05.626005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:06.416841 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:06.416769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:06.636119 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:06.636085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:06.737371 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:06.737257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:06.837666 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:06.837632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:07.031519 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:07.031488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:07.159420 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:07.159384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:07.159564 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:07.159436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:07.159564 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:07.159513 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 23:20:07.159564 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:07.159530 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 23:20:07.159676 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:07.159582 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:15.159567227 +0000 UTC m=+257.957531829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : secret "insights-runtime-extractor-tls" not found Jul 27 23:20:07.159676 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:07.159594 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm podName:b95822c6-eeb5-4ccc-860b-110f03a0d0cc nodeName:}" failed. No retries permitted until 2026-07-27 23:20:15.159588633 +0000 UTC m=+257.957553236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4whwh" (UID: "b95822c6-eeb5-4ccc-860b-110f03a0d0cc") : configmap "kube-rbac-proxy" not found Jul 27 23:20:07.228630 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:07.228604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:07.439532 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:07.439422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:07.825014 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:07.824982 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:08.469723 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:08.469680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:20:08.470132 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:08.469869 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 23:20:08.470132 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:08.469946 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs podName:93e62be3-2c13-473b-b298-ae8f2dde91b8 nodeName:}" failed. No retries permitted until 2026-07-27 23:22:10.469928691 +0000 UTC m=+373.267893292 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs") pod "network-metrics-daemon-l6jq8" (UID: "93e62be3-2c13-473b-b298-ae8f2dde91b8") : secret "metrics-daemon-secret" not found Jul 27 23:20:08.630038 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:08.629998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:09.040628 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:09.040592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:09.231233 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:09.231198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:09.458490 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:09.458409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:09.513524 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:09.513491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:09.729630 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:09.729549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:09.830316 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:09.830277 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:10.427380 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.427345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:10.585612 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.585579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:20:10.588091 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.588070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3fa7c20-89da-424a-9c08-7053736e0cf6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slmlm\" (UID: \"e3fa7c20-89da-424a-9c08-7053736e0cf6\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:20:10.625965 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.625940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h5f97\"" Jul 27 23:20:10.634704 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.634689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" Jul 27 23:20:10.686475 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.686442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:10.686625 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.686488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:20:10.686625 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.686535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:10.686908 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:10.686887 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 23:20:10.687075 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:10.687063 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert podName:dcf565a0-b02a-4958-9786-02ce54f8c298 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:26.687041799 +0000 UTC m=+269.485006408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-5q72l" (UID: "dcf565a0-b02a-4958-9786-02ce54f8c298") : secret "networking-console-plugin-cert" not found Jul 27 23:20:10.687282 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.687259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-service-ca-bundle\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:10.689095 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.689069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5-metrics-certs\") pod \"router-default-74f9f647d5-fbgwn\" (UID: \"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5\") " pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:10.737765 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.737744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vt5xb\"" Jul 27 23:20:10.745628 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.745600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:10.755521 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.755500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm"] Jul 27 23:20:10.758269 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:10.758245 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fa7c20_89da_424a_9c08_7053736e0cf6.slice/crio-fc2fb948c05740be2caf74ef40bc4f72b64b0ddc939f3c3afaec59206c750383 WatchSource:0}: Error finding container fc2fb948c05740be2caf74ef40bc4f72b64b0ddc939f3c3afaec59206c750383: Status 404 returned error can't find the container with id fc2fb948c05740be2caf74ef40bc4f72b64b0ddc939f3c3afaec59206c750383 Jul 27 23:20:10.852156 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.852126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:10.871348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:10.871326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74f9f647d5-fbgwn"] Jul 27 23:20:10.873344 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:10.873319 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ec3ab0_b39f_4241_b2b4_61e84e7b4ad5.slice/crio-00ee932c13d5834fbf389a1b596998998c69945204149364b576e0857de54600 WatchSource:0}: Error finding container 00ee932c13d5834fbf389a1b596998998c69945204149364b576e0857de54600: Status 404 returned error can't find the container with id 00ee932c13d5834fbf389a1b596998998c69945204149364b576e0857de54600 Jul 27 23:20:11.226560 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.226505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:11.565707 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.565670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" event={"ID":"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5","Type":"ContainerStarted","Data":"f4f2a34e5d56a4edd3c532b0fab6394581577ca7017ba8346802509f6bcde161"} Jul 27 23:20:11.565923 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.565728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" event={"ID":"26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5","Type":"ContainerStarted","Data":"00ee932c13d5834fbf389a1b596998998c69945204149364b576e0857de54600"} Jul 27 23:20:11.566723 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.566685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" event={"ID":"e3fa7c20-89da-424a-9c08-7053736e0cf6","Type":"ContainerStarted","Data":"fc2fb948c05740be2caf74ef40bc4f72b64b0ddc939f3c3afaec59206c750383"} Jul 27 23:20:11.585131 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.585088 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" podStartSLOduration=17.585074924 podStartE2EDuration="17.585074924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:11.584285589 +0000 UTC m=+254.382250223" watchObservedRunningTime="2026-07-27 23:20:11.585074924 +0000 UTC m=+254.383039804" Jul 27 23:20:11.627931 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.627905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:11.746497 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.746455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:11.749471 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.749448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:11.828060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:11.827986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:12.426607 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:12.426579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:12.526907 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:12.526863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:12.569861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:12.569811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:12.571148 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:12.571124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74f9f647d5-fbgwn" Jul 27 23:20:12.626710 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:12.626672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:12.827907 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:12.827874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:13.223412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.223331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:13.290068 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.290034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:13.368158 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.368124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:13.468570 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.468534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:13.580050 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.580009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" event={"ID":"e3fa7c20-89da-424a-9c08-7053736e0cf6","Type":"ContainerStarted","Data":"debf4b3e19eae5d3c7a932aaf4f98b0e9f822848c8a0ccaab23fea85d6e533ff"} Jul 27 23:20:13.596749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.596695 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slmlm" podStartSLOduration=17.428255299 podStartE2EDuration="19.596679123s" totalImagesPullingTime="2.168423824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:13.596143664 +0000 UTC m=+256.394108310" watchObservedRunningTime="2026-07-27 23:20:13.596679123 +0000 UTC m=+256.394643746" Jul 27 23:20:13.630953 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:13.630921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:14.026334 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:14.026298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:15.225561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.225522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:15.225561 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.225577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:15.226090 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.226068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:15.227968 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.227951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b95822c6-eeb5-4ccc-860b-110f03a0d0cc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4whwh\" (UID: \"b95822c6-eeb5-4ccc-860b-110f03a0d0cc\") " pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:15.275036 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.275005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4whwh" Jul 27 23:20:15.399910 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.399879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4whwh"] Jul 27 23:20:15.403219 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:15.403187 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95822c6_eeb5_4ccc_860b_110f03a0d0cc.slice/crio-ab7e77f0668370325f5f7029e199608003318cdf293c26fc37a710f605219f50 WatchSource:0}: Error finding container ab7e77f0668370325f5f7029e199608003318cdf293c26fc37a710f605219f50: Status 404 returned error can't find the container with id ab7e77f0668370325f5f7029e199608003318cdf293c26fc37a710f605219f50 Jul 27 23:20:15.587126 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:15.587094 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4whwh" event={"ID":"b95822c6-eeb5-4ccc-860b-110f03a0d0cc","Type":"ContainerStarted","Data":"ab7e77f0668370325f5f7029e199608003318cdf293c26fc37a710f605219f50"} Jul 27 23:20:17.991637 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:17.991592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.092793 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.092756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.192869 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.192840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.491920 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.491889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.664218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.664121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.730831 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.730774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.885235 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.885201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:18.931087 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:18.931039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:19.069111 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:19.069077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:19.702798 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:19.702765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:19.868932 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:19.868897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:20.548939 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:20.548896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:20.608907 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:20.608871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4whwh" event={"ID":"b95822c6-eeb5-4ccc-860b-110f03a0d0cc","Type":"ContainerStarted","Data":"0c11c096590e21734a140654e3adaa861198680a7821067b7271f973cac2b4aa"} Jul 27 23:20:20.685973 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:20.685889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:20.731832 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:20.731796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:24.295797 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:24.295770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:24.394860 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:24.394762 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:24.627886 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:24.627849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4whwh" event={"ID":"b95822c6-eeb5-4ccc-860b-110f03a0d0cc","Type":"ContainerStarted","Data":"9b38af1f38dbd55e5908297ba6930a6281be87240aba0a6869d7a4f2605b5fdc"} Jul 27 23:20:26.722291 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:26.722253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:20:26.724853 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:26.724830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dcf565a0-b02a-4958-9786-02ce54f8c298-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-5q72l\" (UID: \"dcf565a0-b02a-4958-9786-02ce54f8c298\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:20:26.917183 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:26.917151 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5fdqh\"" Jul 27 23:20:26.924970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:26.924945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" Jul 27 23:20:27.402993 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.402961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:27.436895 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.436861 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6f5rt"] Jul 27 23:20:27.453069 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.453022 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6f5rt"] Jul 27 23:20:27.453206 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.453124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.456635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.456610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jmsbf\"" Jul 27 23:20:27.456759 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.456734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 27 23:20:27.457009 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.456841 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 23:20:27.457009 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.456953 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 27 23:20:27.471585 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.471562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:27.528991 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.528961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a5b23-8877-4788-8e63-efecbac839b6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.529099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.529012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab9a5b23-8877-4788-8e63-efecbac839b6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.529099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.529044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkm2x\" (UniqueName: \"kubernetes.io/projected/ab9a5b23-8877-4788-8e63-efecbac839b6-kube-api-access-bkm2x\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.529099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.529083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab9a5b23-8877-4788-8e63-efecbac839b6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.599164 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.599141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:27.630911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.630405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab9a5b23-8877-4788-8e63-efecbac839b6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.630911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.630498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a5b23-8877-4788-8e63-efecbac839b6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.630911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.630535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab9a5b23-8877-4788-8e63-efecbac839b6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.630911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.630578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkm2x\" (UniqueName: \"kubernetes.io/projected/ab9a5b23-8877-4788-8e63-efecbac839b6-kube-api-access-bkm2x\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.631601 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.631546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a5b23-8877-4788-8e63-efecbac839b6-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.634168 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.633631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ab9a5b23-8877-4788-8e63-efecbac839b6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.641891 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.641275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab9a5b23-8877-4788-8e63-efecbac839b6-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.654491 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.654414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkm2x\" (UniqueName: \"kubernetes.io/projected/ab9a5b23-8877-4788-8e63-efecbac839b6-kube-api-access-bkm2x\") pod \"prometheus-operator-84d69f794d-6f5rt\" (UID: \"ab9a5b23-8877-4788-8e63-efecbac839b6\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.727050 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.727031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-5q72l"] Jul 27 23:20:27.729719 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:27.729689 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf565a0_b02a_4958_9786_02ce54f8c298.slice/crio-f45812d0ad41a7bb488ec39a0f78094966ffb3a34dd0e508487c3c38fb34433e WatchSource:0}: Error finding container f45812d0ad41a7bb488ec39a0f78094966ffb3a34dd0e508487c3c38fb34433e: Status 404 returned error can't find the container with id f45812d0ad41a7bb488ec39a0f78094966ffb3a34dd0e508487c3c38fb34433e Jul 27 23:20:27.763798 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.763777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" Jul 27 23:20:27.885930 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.885907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-6f5rt"] Jul 27 23:20:27.888110 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:27.888087 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9a5b23_8877_4788_8e63_efecbac839b6.slice/crio-461214f3ef2e12787ec85acbbc83d66accb6d638ce4477766bacd5e02a696801 WatchSource:0}: Error finding container 461214f3ef2e12787ec85acbbc83d66accb6d638ce4477766bacd5e02a696801: Status 404 returned error can't find the container with id 461214f3ef2e12787ec85acbbc83d66accb6d638ce4477766bacd5e02a696801 Jul 27 23:20:27.895256 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.895235 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59b76c89cc-nqq2v"] Jul 27 23:20:27.899402 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.899387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:27.901483 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901456 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 23:20:27.901635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901616 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 23:20:27.901724 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901623 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 23:20:27.901724 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901639 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 23:20:27.901724 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901692 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 23:20:27.901894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901770 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 23:20:27.901894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901787 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 23:20:27.901894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.901842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8sgz6\"" Jul 27 23:20:27.905661 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:27.905607 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b76c89cc-nqq2v"] Jul 27 23:20:28.033793 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.033767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-config\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.033977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.033809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-serving-cert\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.033977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.033883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-oauth-config\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.034080 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.033972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-service-ca\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.034080 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.034010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj65x\" (UniqueName: \"kubernetes.io/projected/b2d920ed-b61c-456f-87ad-5ba6dfd56339-kube-api-access-sj65x\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.034080 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.034031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-oauth-serving-cert\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.135351 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.135306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-config\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.135351 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.135359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-serving-cert\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.135603 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.135514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-oauth-config\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.135657 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.135601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-service-ca\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.135657 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.135640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj65x\" (UniqueName: \"kubernetes.io/projected/b2d920ed-b61c-456f-87ad-5ba6dfd56339-kube-api-access-sj65x\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.135760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.135676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-oauth-serving-cert\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.136191 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.136108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-config\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.136311 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.136285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-service-ca\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.136373 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.136348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-oauth-serving-cert\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.137959 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.137938 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-serving-cert\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.138042 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.137967 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-oauth-config\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.144123 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.144107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj65x\" (UniqueName: \"kubernetes.io/projected/b2d920ed-b61c-456f-87ad-5ba6dfd56339-kube-api-access-sj65x\") pod \"console-59b76c89cc-nqq2v\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.208977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.208905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:28.329831 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.329783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:28.335710 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.335689 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b76c89cc-nqq2v"] Jul 27 23:20:28.337930 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:28.337903 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d920ed_b61c_456f_87ad_5ba6dfd56339.slice/crio-04bac9ba16bef63cfafb06aa21f13cd388fdda123537bc73d1e0b8402b874e40 WatchSource:0}: Error finding container 04bac9ba16bef63cfafb06aa21f13cd388fdda123537bc73d1e0b8402b874e40: Status 404 returned error can't find the container with id 04bac9ba16bef63cfafb06aa21f13cd388fdda123537bc73d1e0b8402b874e40 Jul 27 23:20:28.655571 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.655541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" event={"ID":"dcf565a0-b02a-4958-9786-02ce54f8c298","Type":"ContainerStarted","Data":"f45812d0ad41a7bb488ec39a0f78094966ffb3a34dd0e508487c3c38fb34433e"} Jul 27 23:20:28.656788 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.656756 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b76c89cc-nqq2v" event={"ID":"b2d920ed-b61c-456f-87ad-5ba6dfd56339","Type":"ContainerStarted","Data":"04bac9ba16bef63cfafb06aa21f13cd388fdda123537bc73d1e0b8402b874e40"} Jul 27 23:20:28.657994 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.657971 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" event={"ID":"ab9a5b23-8877-4788-8e63-efecbac839b6","Type":"ContainerStarted","Data":"461214f3ef2e12787ec85acbbc83d66accb6d638ce4477766bacd5e02a696801"} Jul 27 23:20:28.660239 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.660216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4whwh" event={"ID":"b95822c6-eeb5-4ccc-860b-110f03a0d0cc","Type":"ContainerStarted","Data":"74576654034a989dd19a05cc4b9e370f02742559666030aa327f8c4a5205aa87"} Jul 27 23:20:28.676709 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:28.676659 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4whwh" podStartSLOduration=17.546634287 podStartE2EDuration="29.676643172s" totalImagesPullingTime="12.130008885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 23:19:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:28.675526539 +0000 UTC m=+271.473491167" watchObservedRunningTime="2026-07-27 23:20:28.676643172 +0000 UTC m=+271.474607797" Jul 27 23:20:29.439172 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:29.439131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:29.539377 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:29.539309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:29.697434 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:29.697408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:29.798049 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:29.797933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:30.020639 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.020604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:30.671473 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.671437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" event={"ID":"ab9a5b23-8877-4788-8e63-efecbac839b6","Type":"ContainerStarted","Data":"9fcbedc97fb29034c5f227796b48e7cfd9c0468f65ec7c77aa8e9b12bc7cf3ed"} Jul 27 23:20:30.671955 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.671507 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" event={"ID":"ab9a5b23-8877-4788-8e63-efecbac839b6","Type":"ContainerStarted","Data":"e5a2c08891defdccb46a7b5f717d119981bbdecb90a76380a10ba0d4b60fbe6e"} Jul 27 23:20:30.673161 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.673131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" event={"ID":"dcf565a0-b02a-4958-9786-02ce54f8c298","Type":"ContainerStarted","Data":"b34ad14e270422dcd2c79ab87bc143fbe8b2fbb5ddace5c0f5f2e1d01774cbbf"} Jul 27 23:20:30.685530 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.685480 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-6f5rt" podStartSLOduration=1.825711592 podStartE2EDuration="3.685464601s" totalImagesPullingTime="1.859753009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:30.685403873 +0000 UTC m=+273.483368496" watchObservedRunningTime="2026-07-27 23:20:30.685464601 +0000 UTC m=+273.483429229" Jul 27 23:20:30.700932 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.700886 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-5q72l" podStartSLOduration=34.735899334 podStartE2EDuration="36.700867425s" totalImagesPullingTime="1.964968091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:30.700181439 +0000 UTC m=+273.498146065" watchObservedRunningTime="2026-07-27 23:20:30.700867425 +0000 UTC m=+273.498832051" Jul 27 23:20:30.784403 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.784369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:30.834837 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:30.834790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:31.564718 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:31.564679 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:31.662782 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:31.662752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:32.515564 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.515531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:32.682347 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.682314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b76c89cc-nqq2v" event={"ID":"b2d920ed-b61c-456f-87ad-5ba6dfd56339","Type":"ContainerStarted","Data":"6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1"} Jul 27 23:20:32.697847 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.697780 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59b76c89cc-nqq2v" podStartSLOduration=2.275772885 podStartE2EDuration="5.697760408s" totalImagesPullingTime="3.421987523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:32.696781147 +0000 UTC m=+275.494745772" watchObservedRunningTime="2026-07-27 23:20:32.697760408 +0000 UTC m=+275.495725029" Jul 27 23:20:32.733584 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.733560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:32.770287 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.770212 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l8rpt"] Jul 27 23:20:32.821401 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.821378 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.824158 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.824136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 23:20:32.824272 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.824196 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 23:20:32.824272 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.824229 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2gwgm\"" Jul 27 23:20:32.824272 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.824195 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 23:20:32.833913 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.833895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:32.886145 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-tls\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-wtmp\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-sys\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-textfile\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8c858-45de-4c16-bef7-7ad8222d5832-metrics-client-ca\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-root\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.886604 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.886461 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7wx\" (UniqueName: \"kubernetes.io/projected/7ad8c858-45de-4c16-bef7-7ad8222d5832-kube-api-access-np7wx\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.933327 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.933301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:32.987199 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7wx\" (UniqueName: \"kubernetes.io/projected/7ad8c858-45de-4c16-bef7-7ad8222d5832-kube-api-access-np7wx\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-tls\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-wtmp\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-sys\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987306 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-textfile\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987499 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:32.987333 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 27 23:20:32.987499 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-sys\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987499 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:32.987394 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-tls podName:7ad8c858-45de-4c16-bef7-7ad8222d5832 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:33.487374797 +0000 UTC m=+276.285339406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-tls") pod "node-exporter-l8rpt" (UID: "7ad8c858-45de-4c16-bef7-7ad8222d5832") : secret "node-exporter-tls" not found Jul 27 23:20:32.987499 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8c858-45de-4c16-bef7-7ad8222d5832-metrics-client-ca\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987499 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-wtmp\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987499 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987838 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987537 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-textfile\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987838 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987838 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-root\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.987838 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7ad8c858-45de-4c16-bef7-7ad8222d5832-root\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.988021 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-accelerators-collector-config\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.988021 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.987923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8c858-45de-4c16-bef7-7ad8222d5832-metrics-client-ca\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.990685 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.990661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:32.997293 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:32.997267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7wx\" (UniqueName: \"kubernetes.io/projected/7ad8c858-45de-4c16-bef7-7ad8222d5832-kube-api-access-np7wx\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:33.032884 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.032781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.142050 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.138048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.205601 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.205565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.305479 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.305393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.405639 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.405601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.454522 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.454493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.491461 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.491434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-tls\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:33.493940 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.493923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7ad8c858-45de-4c16-bef7-7ad8222d5832-node-exporter-tls\") pod \"node-exporter-l8rpt\" (UID: \"7ad8c858-45de-4c16-bef7-7ad8222d5832\") " pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:33.733652 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.733582 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l8rpt" Jul 27 23:20:33.737981 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.737950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.797557 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.797528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:33.820010 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:33.819964 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad8c858_45de_4c16_bef7_7ad8222d5832.slice/crio-43604d1d09deee12446cc72cd90beea3b463d4f2b4a4fcf8da5e3fa913126265 WatchSource:0}: Error finding container 43604d1d09deee12446cc72cd90beea3b463d4f2b4a4fcf8da5e3fa913126265: Status 404 returned error can't find the container with id 43604d1d09deee12446cc72cd90beea3b463d4f2b4a4fcf8da5e3fa913126265 Jul 27 23:20:33.939082 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:33.939035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:34.692698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:34.692659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8rpt" event={"ID":"7ad8c858-45de-4c16-bef7-7ad8222d5832","Type":"ContainerStarted","Data":"43604d1d09deee12446cc72cd90beea3b463d4f2b4a4fcf8da5e3fa913126265"} Jul 27 23:20:35.697811 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.697777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:35.699727 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.699695 2564 generic.go:356] "Generic (PLEG): container finished" podID="7ad8c858-45de-4c16-bef7-7ad8222d5832" containerID="192030b3af72d22d36454a8fa03d2b55cb524d20992f6838a7f667926965feba" exitCode=0 Jul 27 23:20:35.699904 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.699791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8rpt" event={"ID":"7ad8c858-45de-4c16-bef7-7ad8222d5832","Type":"ContainerDied","Data":"192030b3af72d22d36454a8fa03d2b55cb524d20992f6838a7f667926965feba"} Jul 27 23:20:35.749648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.749625 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69b8849666-jn92r"] Jul 27 23:20:35.766151 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.766128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:35.776104 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.776083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.778299 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.778279 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 27 23:20:35.778554 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.778533 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 27 23:20:35.778781 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.778761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-46mlqfjl9pbtr\"" Jul 27 23:20:35.778900 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.778810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 27 23:20:35.779005 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.778282 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 27 23:20:35.779065 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.778771 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 27 23:20:35.779564 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.779319 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rppmh\"" Jul 27 23:20:35.781743 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.781722 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69b8849666-jn92r"] Jul 27 23:20:35.913653 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.913764 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.913764 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8z6\" (UniqueName: \"kubernetes.io/projected/72a822a8-7799-4a0c-95a7-e6978f9dcc51-kube-api-access-rj8z6\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.913764 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72a822a8-7799-4a0c-95a7-e6978f9dcc51-metrics-client-ca\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.913954 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.913954 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-tls\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.913954 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.914090 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.913966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-grpc-tls\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:35.919103 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:35.919084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:36.014476 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-tls\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014632 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014632 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-grpc-tls\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014632 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014763 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014763 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8z6\" (UniqueName: \"kubernetes.io/projected/72a822a8-7799-4a0c-95a7-e6978f9dcc51-kube-api-access-rj8z6\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014763 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72a822a8-7799-4a0c-95a7-e6978f9dcc51-metrics-client-ca\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.014763 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.014759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.015937 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.015877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/72a822a8-7799-4a0c-95a7-e6978f9dcc51-metrics-client-ca\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.017594 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.017568 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-tls\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.017766 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.017741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.017875 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.017858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-grpc-tls\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.017941 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.017892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.017981 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.017966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.018014 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.017982 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/72a822a8-7799-4a0c-95a7-e6978f9dcc51-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.019349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.019326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:36.021551 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.021532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8z6\" (UniqueName: \"kubernetes.io/projected/72a822a8-7799-4a0c-95a7-e6978f9dcc51-kube-api-access-rj8z6\") pod \"thanos-querier-69b8849666-jn92r\" (UID: \"72a822a8-7799-4a0c-95a7-e6978f9dcc51\") " pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.088911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.088888 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:36.241090 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.240804 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69b8849666-jn92r"] Jul 27 23:20:36.243252 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:36.243225 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a822a8_7799_4a0c_95a7_e6978f9dcc51.slice/crio-cbdf871a250d1e811c363c67f3c300adfb4701941592c316dc99a5aa34ed1b8d WatchSource:0}: Error finding container cbdf871a250d1e811c363c67f3c300adfb4701941592c316dc99a5aa34ed1b8d: Status 404 returned error can't find the container with id cbdf871a250d1e811c363c67f3c300adfb4701941592c316dc99a5aa34ed1b8d Jul 27 23:20:36.475868 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.475775 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b46b45f6b-f5rbm"] Jul 27 23:20:36.502040 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.502018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b46b45f6b-f5rbm"] Jul 27 23:20:36.502180 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.502109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.513367 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.513325 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 23:20:36.619949 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.619924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4msw\" (UniqueName: \"kubernetes.io/projected/58a740e4-2f35-4d48-9009-3e24d9408473-kube-api-access-l4msw\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.620086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.619958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-console-config\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.620086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.619992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-trusted-ca-bundle\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.620086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.620018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-serving-cert\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.620086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.620062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-oauth-serving-cert\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.620086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.620077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-oauth-config\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.620254 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.620151 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-service-ca\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.704365 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.704335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"cbdf871a250d1e811c363c67f3c300adfb4701941592c316dc99a5aa34ed1b8d"} Jul 27 23:20:36.706182 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.706160 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8rpt" event={"ID":"7ad8c858-45de-4c16-bef7-7ad8222d5832","Type":"ContainerStarted","Data":"2c6947ce38797181f8deddcf29f099a4376610c1b08fd825e703fde44fe7e3d8"} Jul 27 23:20:36.706291 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.706204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l8rpt" event={"ID":"7ad8c858-45de-4c16-bef7-7ad8222d5832","Type":"ContainerStarted","Data":"8c22b1e120b87f3fcb49d718758b993fd627edfc4605315e08bf17b05089c4a8"} Jul 27 23:20:36.721013 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.720988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-trusted-ca-bundle\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721107 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-serving-cert\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721107 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-oauth-serving-cert\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721107 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-oauth-config\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721269 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-service-ca\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721269 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4msw\" (UniqueName: \"kubernetes.io/projected/58a740e4-2f35-4d48-9009-3e24d9408473-kube-api-access-l4msw\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721269 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-console-config\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.721955 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.721933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-oauth-serving-cert\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.722088 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.722019 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-service-ca\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.722239 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.722216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-console-config\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.722719 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.722700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-trusted-ca-bundle\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.723022 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.722982 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l8rpt" podStartSLOduration=3.361425189 podStartE2EDuration="4.722968229s" totalImagesPullingTime="1.306092893s" totalInitContainerRuntime="55.450147ms" isStatefulPod=true podCreationTimestamp="2026-07-27 23:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:36.722135852 +0000 UTC m=+279.520100476" watchObservedRunningTime="2026-07-27 23:20:36.722968229 +0000 UTC m=+279.520932879" Jul 27 23:20:36.723797 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.723779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-oauth-config\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.724661 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.724637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-serving-cert\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.729299 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.729245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4msw\" (UniqueName: \"kubernetes.io/projected/58a740e4-2f35-4d48-9009-3e24d9408473-kube-api-access-l4msw\") pod \"console-7b46b45f6b-f5rbm\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.815870 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.815843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:36.966708 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:36.966680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b46b45f6b-f5rbm"] Jul 27 23:20:36.971153 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:36.971123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a740e4_2f35_4d48_9009_3e24d9408473.slice/crio-d70c3860109faefd4a11d6e3584d6f3e3663447a2426e81e601723bde5b77ce0 WatchSource:0}: Error finding container d70c3860109faefd4a11d6e3584d6f3e3663447a2426e81e601723bde5b77ce0: Status 404 returned error can't find the container with id d70c3860109faefd4a11d6e3584d6f3e3663447a2426e81e601723bde5b77ce0 Jul 27 23:20:37.059186 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.059153 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8c94c6655-bjth8"] Jul 27 23:20:37.073034 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.072867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8c94c6655-bjth8"] Jul 27 23:20:37.073034 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.072975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.075129 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.075105 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 27 23:20:37.075621 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.075469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-267tv\"" Jul 27 23:20:37.075765 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.075746 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 27 23:20:37.075851 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.075773 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cpmb1ibl6b571\"" Jul 27 23:20:37.075915 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.075854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 27 23:20:37.076083 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.076064 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 27 23:20:37.093413 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.093387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:37.174391 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.174357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:37.227627 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-secret-metrics-server-client-certs\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.227809 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-client-ca-bundle\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.227809 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c339d75-f668-4f72-b9e3-d7ced388b27e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.227809 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jlk\" (UniqueName: \"kubernetes.io/projected/2c339d75-f668-4f72-b9e3-d7ced388b27e-kube-api-access-77jlk\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.227971 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-secret-metrics-server-tls\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.227971 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2c339d75-f668-4f72-b9e3-d7ced388b27e-metrics-server-audit-profiles\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.227971 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.227957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2c339d75-f668-4f72-b9e3-d7ced388b27e-audit-log\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329201 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-secret-metrics-server-tls\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329201 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2c339d75-f668-4f72-b9e3-d7ced388b27e-metrics-server-audit-profiles\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2c339d75-f668-4f72-b9e3-d7ced388b27e-audit-log\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-secret-metrics-server-client-certs\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329426 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-client-ca-bundle\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329469 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c339d75-f668-4f72-b9e3-d7ced388b27e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329469 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jlk\" (UniqueName: \"kubernetes.io/projected/2c339d75-f668-4f72-b9e3-d7ced388b27e-kube-api-access-77jlk\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.329708 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.329675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2c339d75-f668-4f72-b9e3-d7ced388b27e-audit-log\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.330201 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.330166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2c339d75-f668-4f72-b9e3-d7ced388b27e-metrics-server-audit-profiles\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.330323 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.330297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c339d75-f668-4f72-b9e3-d7ced388b27e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.331759 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.331732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-secret-metrics-server-client-certs\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.332020 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.331998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-secret-metrics-server-tls\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.332081 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.332024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2c339d75-f668-4f72-b9e3-d7ced388b27e-client-ca-bundle\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.337135 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.337110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jlk\" (UniqueName: \"kubernetes.io/projected/2c339d75-f668-4f72-b9e3-d7ced388b27e-kube-api-access-77jlk\") pod \"metrics-server-8c94c6655-bjth8\" (UID: \"2c339d75-f668-4f72-b9e3-d7ced388b27e\") " pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.385614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.385543 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:37.538905 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.538872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:37.540333 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.539914 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf"] Jul 27 23:20:37.556448 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:37.556420 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c339d75_f668_4f72_b9e3_d7ced388b27e.slice/crio-5a684d4fb1a4196533cb781dd60c1e11a5724214ea946d65422b441e0f626914 WatchSource:0}: Error finding container 5a684d4fb1a4196533cb781dd60c1e11a5724214ea946d65422b441e0f626914: Status 404 returned error can't find the container with id 5a684d4fb1a4196533cb781dd60c1e11a5724214ea946d65422b441e0f626914 Jul 27 23:20:37.558567 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.558528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf"] Jul 27 23:20:37.558567 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.558563 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8c94c6655-bjth8"] Jul 27 23:20:37.558764 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.558653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:37.560886 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.560855 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 27 23:20:37.560986 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.560894 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tjx5g\"" Jul 27 23:20:37.592554 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.592498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:37.711785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.711744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" event={"ID":"2c339d75-f668-4f72-b9e3-d7ced388b27e","Type":"ContainerStarted","Data":"5a684d4fb1a4196533cb781dd60c1e11a5724214ea946d65422b441e0f626914"} Jul 27 23:20:37.716315 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.716264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b46b45f6b-f5rbm" event={"ID":"58a740e4-2f35-4d48-9009-3e24d9408473","Type":"ContainerStarted","Data":"c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4"} Jul 27 23:20:37.716463 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.716322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b46b45f6b-f5rbm" event={"ID":"58a740e4-2f35-4d48-9009-3e24d9408473","Type":"ContainerStarted","Data":"d70c3860109faefd4a11d6e3584d6f3e3663447a2426e81e601723bde5b77ce0"} Jul 27 23:20:37.732249 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.732210 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b46b45f6b-f5rbm" podStartSLOduration=1.732195444 podStartE2EDuration="1.732195444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:37.731075798 +0000 UTC m=+280.529040420" watchObservedRunningTime="2026-07-27 23:20:37.732195444 +0000 UTC m=+280.530160068" Jul 27 23:20:37.734328 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.734301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:37.734420 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.734382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdz25\" (UniqueName: \"kubernetes.io/projected/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-kube-api-access-pdz25\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:37.835424 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.835384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdz25\" (UniqueName: \"kubernetes.io/projected/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-kube-api-access-pdz25\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:37.835748 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.835729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:37.835898 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:37.835880 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 27 23:20:37.835973 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:37.835947 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-monitoring-plugin-cert podName:bc6f33c0-4bd1-42bb-be29-ebaa8af21806 nodeName:}" failed. No retries permitted until 2026-07-27 23:20:38.335927116 +0000 UTC m=+281.133891723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-fnnkf" (UID: "bc6f33c0-4bd1-42bb-be29-ebaa8af21806") : secret "monitoring-plugin-cert" not found Jul 27 23:20:37.844057 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:37.844013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdz25\" (UniqueName: \"kubernetes.io/projected/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-kube-api-access-pdz25\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:38.086527 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.086489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:38.209950 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.209872 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:38.209950 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.209937 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:20:38.211591 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.211552 2564 patch_prober.go:28] interesting pod/console-59b76c89cc-nqq2v container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body= Jul 27 23:20:38.211721 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.211598 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-59b76c89cc-nqq2v" podUID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" containerName="console" probeResult="failure" output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" Jul 27 23:20:38.340128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.340088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:38.343349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.343319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc6f33c0-4bd1-42bb-be29-ebaa8af21806-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-fnnkf\" (UID: \"bc6f33c0-4bd1-42bb-be29-ebaa8af21806\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:38.471697 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.471626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:38.607748 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.607672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf"] Jul 27 23:20:38.609855 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:38.609811 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6f33c0_4bd1_42bb_be29_ebaa8af21806.slice/crio-02a4346f1906810166a4c04e7bfff5af4bcdbf783b01cca67f9eba14e5ce062e WatchSource:0}: Error finding container 02a4346f1906810166a4c04e7bfff5af4bcdbf783b01cca67f9eba14e5ce062e: Status 404 returned error can't find the container with id 02a4346f1906810166a4c04e7bfff5af4bcdbf783b01cca67f9eba14e5ce062e Jul 27 23:20:38.722495 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.722384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" event={"ID":"bc6f33c0-4bd1-42bb-be29-ebaa8af21806","Type":"ContainerStarted","Data":"02a4346f1906810166a4c04e7bfff5af4bcdbf783b01cca67f9eba14e5ce062e"} Jul 27 23:20:38.728448 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.727034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"612c6f01a8f61cd224dacb122f9d8124629072887017ac7a0a5abd2e98b2fe14"} Jul 27 23:20:38.948801 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:38.948742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:39.734393 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:39.734346 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"441a75ff9896e67df7f1f9856ec986eddeac689cd029ab4f97ee8ce4895b6d10"} Jul 27 23:20:39.734786 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:39.734444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"4311961b3681906086fdc69e301c264414c1465a06883bdf39607a5b062470ba"} Jul 27 23:20:40.741041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.741000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"e5bf0c3827d9db86a5d04f11d3c16f057a1f3cec7de248dfa0cefb3358097ef7"} Jul 27 23:20:40.741478 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.741075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"007a4eabc826289c71006012941f8cb4c7f9f4ce44470893875ca175aa7c3d2a"} Jul 27 23:20:40.741478 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.741124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" event={"ID":"72a822a8-7799-4a0c-95a7-e6978f9dcc51","Type":"ContainerStarted","Data":"712ff54a4ca346e7a38da8cacb5b05095bf275728ddd140d2fe32b1bb8b73b84"} Jul 27 23:20:40.741478 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.741171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:40.742551 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.742530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" event={"ID":"bc6f33c0-4bd1-42bb-be29-ebaa8af21806","Type":"ContainerStarted","Data":"3e429f98f42058b4addbd7658ab762701292125716673ae3ec392cc2291ade67"} Jul 27 23:20:40.742691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.742668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:40.744140 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.744113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" event={"ID":"2c339d75-f668-4f72-b9e3-d7ced388b27e","Type":"ContainerStarted","Data":"8ef2410232586d03f909a10f4997c0233619f63c71cb213cbad00072d2984047"} Jul 27 23:20:40.748244 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.748226 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" Jul 27 23:20:40.762856 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.762795 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" podStartSLOduration=1.940705508 podStartE2EDuration="5.762781489s" totalImagesPullingTime="3.822075981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:40.762377903 +0000 UTC m=+283.560342527" watchObservedRunningTime="2026-07-27 23:20:40.762781489 +0000 UTC m=+283.560746112" Jul 27 23:20:40.776910 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.776805 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" podStartSLOduration=1.072905498 podStartE2EDuration="3.77678968s" totalImagesPullingTime="2.703884182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:40.776249901 +0000 UTC m=+283.574214526" watchObservedRunningTime="2026-07-27 23:20:40.77678968 +0000 UTC m=+283.574754305" Jul 27 23:20:40.788663 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:40.788623 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-fnnkf" podStartSLOduration=2.137383483 podStartE2EDuration="3.788610073s" totalImagesPullingTime="1.65122659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:40.788088627 +0000 UTC m=+283.586053250" watchObservedRunningTime="2026-07-27 23:20:40.788610073 +0000 UTC m=+283.586574696" Jul 27 23:20:41.981296 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:41.981247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:20:41.981296 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:41.981299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:20:41.983725 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:41.983704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3deae1b-7472-44a8-881d-26b065db3335-metrics-tls\") pod \"dns-default-rlfmr\" (UID: \"e3deae1b-7472-44a8-881d-26b065db3335\") " pod="openshift-dns/dns-default-rlfmr" Jul 27 23:20:41.983846 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:41.983803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"image-registry-6cd8dc995c-frmhn\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:20:42.098948 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.098918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:42.199443 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.199411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:42.280522 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.280480 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-66lfc\"" Jul 27 23:20:42.280522 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.280479 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2qdw\"" Jul 27 23:20:42.288359 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.288338 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rlfmr" Jul 27 23:20:42.288423 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.288369 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:20:42.433272 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.433240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cd8dc995c-frmhn"] Jul 27 23:20:42.436740 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:42.436711 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560af22b_0e13_4e3b_b21f_c02c8f6011f6.slice/crio-c42ed72510ed3ebaa79d2b32c452ad83ea0f4baa1ed77ed140467f806e62091c WatchSource:0}: Error finding container c42ed72510ed3ebaa79d2b32c452ad83ea0f4baa1ed77ed140467f806e62091c: Status 404 returned error can't find the container with id c42ed72510ed3ebaa79d2b32c452ad83ea0f4baa1ed77ed140467f806e62091c Jul 27 23:20:42.445901 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.445883 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rlfmr"] Jul 27 23:20:42.449007 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:42.448985 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3deae1b_7472_44a8_881d_26b065db3335.slice/crio-5e103b4ac89ae5714450fc854acba2205b4fb51b75690d14241c47122968d18a WatchSource:0}: Error finding container 5e103b4ac89ae5714450fc854acba2205b4fb51b75690d14241c47122968d18a: Status 404 returned error can't find the container with id 5e103b4ac89ae5714450fc854acba2205b4fb51b75690d14241c47122968d18a Jul 27 23:20:42.754634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.754598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlfmr" event={"ID":"e3deae1b-7472-44a8-881d-26b065db3335","Type":"ContainerStarted","Data":"5e103b4ac89ae5714450fc854acba2205b4fb51b75690d14241c47122968d18a"} Jul 27 23:20:42.757301 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.757237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" event={"ID":"560af22b-0e13-4e3b-b21f-c02c8f6011f6","Type":"ContainerStarted","Data":"ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85"} Jul 27 23:20:42.757301 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.757293 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:20:42.757518 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.757313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" event={"ID":"560af22b-0e13-4e3b-b21f-c02c8f6011f6","Type":"ContainerStarted","Data":"c42ed72510ed3ebaa79d2b32c452ad83ea0f4baa1ed77ed140467f806e62091c"} Jul 27 23:20:42.782538 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:42.782480 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" podStartSLOduration=262.782459891 podStartE2EDuration="4m22.782459891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:42.781615997 +0000 UTC m=+285.579580622" watchObservedRunningTime="2026-07-27 23:20:42.782459891 +0000 UTC m=+285.580424517" Jul 27 23:20:43.897971 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:43.897937 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b76c89cc-nqq2v"] Jul 27 23:20:43.931922 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:43.931891 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f8dbd688f-2jlgc"] Jul 27 23:20:43.934194 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:43.934170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:43.960482 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:43.960416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8dbd688f-2jlgc"] Jul 27 23:20:44.102542 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-oauth-serving-cert\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.102725 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-trusted-ca-bundle\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.102810 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-oauth-config\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.102810 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-serving-cert\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.102941 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8p4\" (UniqueName: \"kubernetes.io/projected/f7caaa6f-ec23-4644-9eeb-4cd22947efab-kube-api-access-xg8p4\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.102996 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-config\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.102996 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.102990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-service-ca\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-oauth-config\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-serving-cert\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8p4\" (UniqueName: \"kubernetes.io/projected/f7caaa6f-ec23-4644-9eeb-4cd22947efab-kube-api-access-xg8p4\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-config\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-service-ca\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-oauth-serving-cert\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.203961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.203896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-trusted-ca-bundle\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.204627 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.204592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-config\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.204780 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.204759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-oauth-serving-cert\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.204922 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.204901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-service-ca\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.204977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.204946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-trusted-ca-bundle\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.206774 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.206746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-oauth-config\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.207223 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.207196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-serving-cert\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.211806 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.211784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8p4\" (UniqueName: \"kubernetes.io/projected/f7caaa6f-ec23-4644-9eeb-4cd22947efab-kube-api-access-xg8p4\") pod \"console-5f8dbd688f-2jlgc\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.243679 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.243520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:44.608369 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.608309 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8dbd688f-2jlgc"] Jul 27 23:20:44.611147 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:20:44.611122 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7caaa6f_ec23_4644_9eeb_4cd22947efab.slice/crio-0f01b970ffecdcfea68ca7cf57daa2b17d47e270c846eefa97b8ad8296256f82 WatchSource:0}: Error finding container 0f01b970ffecdcfea68ca7cf57daa2b17d47e270c846eefa97b8ad8296256f82: Status 404 returned error can't find the container with id 0f01b970ffecdcfea68ca7cf57daa2b17d47e270c846eefa97b8ad8296256f82 Jul 27 23:20:44.771786 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.771752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8dbd688f-2jlgc" event={"ID":"f7caaa6f-ec23-4644-9eeb-4cd22947efab","Type":"ContainerStarted","Data":"8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565"} Jul 27 23:20:44.772001 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.771850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8dbd688f-2jlgc" event={"ID":"f7caaa6f-ec23-4644-9eeb-4cd22947efab","Type":"ContainerStarted","Data":"0f01b970ffecdcfea68ca7cf57daa2b17d47e270c846eefa97b8ad8296256f82"} Jul 27 23:20:44.773690 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.773665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlfmr" event={"ID":"e3deae1b-7472-44a8-881d-26b065db3335","Type":"ContainerStarted","Data":"c07c5abb3955d434b79a2b5d4c8962ca5d65f55b162a0c1c0f4add42fca709ed"} Jul 27 23:20:44.773794 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.773705 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rlfmr" event={"ID":"e3deae1b-7472-44a8-881d-26b065db3335","Type":"ContainerStarted","Data":"4b7f3d33e6b41b109190612cf2dda8e7c6638ab1808d2e314c7d3450a803bcba"} Jul 27 23:20:44.773794 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.773786 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rlfmr" Jul 27 23:20:44.789833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.789764 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f8dbd688f-2jlgc" podStartSLOduration=1.7897500910000002 podStartE2EDuration="1.789750091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:44.788940706 +0000 UTC m=+287.586905331" watchObservedRunningTime="2026-07-27 23:20:44.789750091 +0000 UTC m=+287.587714786" Jul 27 23:20:44.806241 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:44.806197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rlfmr" podStartSLOduration=250.784688035 podStartE2EDuration="4m12.806184311s" totalImagesPullingTime="2.021496276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:20:44.805085163 +0000 UTC m=+287.603049789" watchObservedRunningTime="2026-07-27 23:20:44.806184311 +0000 UTC m=+287.604148936" Jul 27 23:20:46.122783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.122745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:46.170933 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.170902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:46.753941 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.753910 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69b8849666-jn92r" Jul 27 23:20:46.816831 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.816780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:46.817017 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.816870 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:20:46.818074 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.818047 2564 patch_prober.go:28] interesting pod/console-7b46b45f6b-f5rbm container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.21:8443/health\": dial tcp 10.132.0.21:8443: connect: connection refused" start-of-body= Jul 27 23:20:46.818180 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:46.818099 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7b46b45f6b-f5rbm" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" containerName="console" probeResult="failure" output="Get \"https://10.132.0.21:8443/health\": dial tcp 10.132.0.21:8443: connect: connection refused" Jul 27 23:20:47.292573 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:47.292543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:47.339569 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:47.339538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:53.778117 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:20:53.778068 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l6jq8" podUID="93e62be3-2c13-473b-b298-ae8f2dde91b8" Jul 27 23:20:54.244629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:54.244537 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:54.244629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:54.244605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:20:54.246000 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:54.245975 2564 patch_prober.go:28] interesting pod/console-5f8dbd688f-2jlgc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.24:8443/health\": dial tcp 10.132.0.24:8443: connect: connection refused" start-of-body= Jul 27 23:20:54.246111 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:54.246015 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5f8dbd688f-2jlgc" podUID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" containerName="console" probeResult="failure" output="Get \"https://10.132.0.24:8443/health\": dial tcp 10.132.0.24:8443: connect: connection refused" Jul 27 23:20:54.780961 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:54.780932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rlfmr" Jul 27 23:20:55.711309 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:55.711276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:55.760019 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:55.759987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:56.381436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:56.381406 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cd8dc995c-frmhn"] Jul 27 23:20:56.451612 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:56.451586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:56.551760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:56.551733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:20:56.816748 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:56.816713 2564 patch_prober.go:28] interesting pod/console-7b46b45f6b-f5rbm container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.21:8443/health\": dial tcp 10.132.0.21:8443: connect: connection refused" start-of-body= Jul 27 23:20:56.816967 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:56.816778 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7b46b45f6b-f5rbm" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" containerName="console" probeResult="failure" output="Get \"https://10.132.0.21:8443/health\": dial tcp 10.132.0.21:8443: connect: connection refused" Jul 27 23:20:57.386469 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:57.386435 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:57.386879 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:57.386507 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:20:57.642162 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:20:57.642097 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 23:21:03.884938 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:03.884896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:04.248275 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:04.248186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:21:04.251978 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:04.251958 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:21:04.304264 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:04.304235 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b46b45f6b-f5rbm"] Jul 27 23:21:05.067650 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:05.067618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:05.289536 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:05.289491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:05.338178 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:05.338100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:06.387291 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:06.387264 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:21:08.776686 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:08.776651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:21:08.923429 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:08.923389 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59b76c89cc-nqq2v" podUID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" containerName="console" containerID="cri-o://6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1" gracePeriod=15 Jul 27 23:21:09.168112 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.168085 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:21:09.292097 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.292015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:09.344549 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.344509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:09.347624 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.347607 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-oauth-serving-cert\") pod \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " Jul 27 23:21:09.347682 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.347646 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj65x\" (UniqueName: \"kubernetes.io/projected/b2d920ed-b61c-456f-87ad-5ba6dfd56339-kube-api-access-sj65x\") pod \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " Jul 27 23:21:09.347731 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.347720 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-config\") pod \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " Jul 27 23:21:09.347774 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.347744 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-serving-cert\") pod \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " Jul 27 23:21:09.347774 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.347765 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-oauth-config\") pod \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " Jul 27 23:21:09.347912 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.347834 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-service-ca\") pod \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\" (UID: \"b2d920ed-b61c-456f-87ad-5ba6dfd56339\") " Jul 27 23:21:09.348100 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.348070 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2d920ed-b61c-456f-87ad-5ba6dfd56339" (UID: "b2d920ed-b61c-456f-87ad-5ba6dfd56339"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:09.348240 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.348106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-config" (OuterVolumeSpecName: "console-config") pod "b2d920ed-b61c-456f-87ad-5ba6dfd56339" (UID: "b2d920ed-b61c-456f-87ad-5ba6dfd56339"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:09.348325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.348297 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2d920ed-b61c-456f-87ad-5ba6dfd56339" (UID: "b2d920ed-b61c-456f-87ad-5ba6dfd56339"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:09.350116 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.350096 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d920ed-b61c-456f-87ad-5ba6dfd56339-kube-api-access-sj65x" (OuterVolumeSpecName: "kube-api-access-sj65x") pod "b2d920ed-b61c-456f-87ad-5ba6dfd56339" (UID: "b2d920ed-b61c-456f-87ad-5ba6dfd56339"). InnerVolumeSpecName "kube-api-access-sj65x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:21:09.350357 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.350340 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2d920ed-b61c-456f-87ad-5ba6dfd56339" (UID: "b2d920ed-b61c-456f-87ad-5ba6dfd56339"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:21:09.350423 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.350371 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2d920ed-b61c-456f-87ad-5ba6dfd56339" (UID: "b2d920ed-b61c-456f-87ad-5ba6dfd56339"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:21:09.448772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.448730 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-oauth-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:09.448772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.448762 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj65x\" (UniqueName: \"kubernetes.io/projected/b2d920ed-b61c-456f-87ad-5ba6dfd56339-kube-api-access-sj65x\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:09.448772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.448776 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:09.449030 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.448788 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:09.449030 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.448800 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2d920ed-b61c-456f-87ad-5ba6dfd56339-console-oauth-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:09.449030 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.448811 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d920ed-b61c-456f-87ad-5ba6dfd56339-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:09.867070 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.867036 2564 generic.go:356] "Generic (PLEG): container finished" podID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" containerID="6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1" exitCode=0 Jul 27 23:21:09.867410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.867080 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b76c89cc-nqq2v" Jul 27 23:21:09.867410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.867078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b76c89cc-nqq2v" event={"ID":"b2d920ed-b61c-456f-87ad-5ba6dfd56339","Type":"ContainerDied","Data":"6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1"} Jul 27 23:21:09.867410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.867193 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b76c89cc-nqq2v" event={"ID":"b2d920ed-b61c-456f-87ad-5ba6dfd56339","Type":"ContainerDied","Data":"04bac9ba16bef63cfafb06aa21f13cd388fdda123537bc73d1e0b8402b874e40"} Jul 27 23:21:09.867410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.867223 2564 scope.go:117] "RemoveContainer" containerID="6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1" Jul 27 23:21:09.875536 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.875521 2564 scope.go:117] "RemoveContainer" containerID="6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1" Jul 27 23:21:09.875799 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:21:09.875780 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1\": container with ID starting with 6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1 not found: ID does not exist" containerID="6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1" Jul 27 23:21:09.875868 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.875808 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1"} err="failed to get container status \"6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1\": rpc error: code = NotFound desc = could not find container \"6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1\": container with ID starting with 6b46eaead2526c353768a4212e3558d757d607a072467201095167d836e538f1 not found: ID does not exist" Jul 27 23:21:09.884176 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.884154 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b76c89cc-nqq2v"] Jul 27 23:21:09.887474 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:09.887456 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59b76c89cc-nqq2v"] Jul 27 23:21:11.781837 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:11.781782 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" path="/var/lib/kubelet/pods/b2d920ed-b61c-456f-87ad-5ba6dfd56339/volumes" Jul 27 23:21:13.885875 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:13.885835 2564 generic.go:356] "Generic (PLEG): container finished" podID="954778d1-cf9f-46d3-87d3-4e75c6c292ed" containerID="4b1145bac0ca6d7e5f5415f98367cc793f7973ff61428268c52fe0d9d2dc9441" exitCode=0 Jul 27 23:21:13.886237 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:13.885895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" event={"ID":"954778d1-cf9f-46d3-87d3-4e75c6c292ed","Type":"ContainerDied","Data":"4b1145bac0ca6d7e5f5415f98367cc793f7973ff61428268c52fe0d9d2dc9441"} Jul 27 23:21:13.886237 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:13.886234 2564 scope.go:117] "RemoveContainer" containerID="4b1145bac0ca6d7e5f5415f98367cc793f7973ff61428268c52fe0d9d2dc9441" Jul 27 23:21:14.891199 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:14.891167 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-kkmc9" event={"ID":"954778d1-cf9f-46d3-87d3-4e75c6c292ed","Type":"ContainerStarted","Data":"b896f27b3e9879dcf7c6dd5b21bc7ba326be1a606f07c522229e689c7fdd7c7a"} Jul 27 23:21:15.875924 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:15.875888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:15.923023 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:15.922993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:17.391467 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:17.391436 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:21:17.395440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:17.395418 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8c94c6655-bjth8" Jul 27 23:21:21.400981 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.400939 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" podUID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" containerName="registry" containerID="cri-o://ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85" gracePeriod=30 Jul 27 23:21:21.650348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.650326 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:21:21.747947 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.747880 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppjl8\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-kube-api-access-ppjl8\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.747947 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.747940 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-trusted-ca\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.747975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-bound-sa-token\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748010 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748033 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-certificates\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748057 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-installation-pull-secrets\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748086 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-image-registry-private-configuration\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/560af22b-0e13-4e3b-b21f-c02c8f6011f6-ca-trust-extracted\") pod \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\" (UID: \"560af22b-0e13-4e3b-b21f-c02c8f6011f6\") " Jul 27 23:21:21.748452 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:21.748918 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.748887 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:21.750796 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.750753 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:21:21.751068 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.750862 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:21:21.751237 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.751215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-kube-api-access-ppjl8" (OuterVolumeSpecName: "kube-api-access-ppjl8") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "kube-api-access-ppjl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:21:21.751345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.751280 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:21:21.751473 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.751453 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:21:21.757385 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.757358 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560af22b-0e13-4e3b-b21f-c02c8f6011f6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "560af22b-0e13-4e3b-b21f-c02c8f6011f6" (UID: "560af22b-0e13-4e3b-b21f-c02c8f6011f6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:21:21.849018 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.848986 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjl8\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-kube-api-access-ppjl8\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849018 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849013 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-trusted-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849026 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-bound-sa-token\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849038 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-tls\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849049 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/560af22b-0e13-4e3b-b21f-c02c8f6011f6-registry-certificates\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849060 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-installation-pull-secrets\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849074 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/560af22b-0e13-4e3b-b21f-c02c8f6011f6-image-registry-private-configuration\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.849187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.849086 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/560af22b-0e13-4e3b-b21f-c02c8f6011f6-ca-trust-extracted\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:21.918348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.918317 2564 generic.go:356] "Generic (PLEG): container finished" podID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" containerID="ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85" exitCode=0 Jul 27 23:21:21.918452 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.918354 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" Jul 27 23:21:21.918452 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.918407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" event={"ID":"560af22b-0e13-4e3b-b21f-c02c8f6011f6","Type":"ContainerDied","Data":"ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85"} Jul 27 23:21:21.918519 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.918457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cd8dc995c-frmhn" event={"ID":"560af22b-0e13-4e3b-b21f-c02c8f6011f6","Type":"ContainerDied","Data":"c42ed72510ed3ebaa79d2b32c452ad83ea0f4baa1ed77ed140467f806e62091c"} Jul 27 23:21:21.918519 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.918479 2564 scope.go:117] "RemoveContainer" containerID="ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85" Jul 27 23:21:21.927160 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.927144 2564 scope.go:117] "RemoveContainer" containerID="ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85" Jul 27 23:21:21.927356 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:21:21.927340 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85\": container with ID starting with ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85 not found: ID does not exist" containerID="ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85" Jul 27 23:21:21.927410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.927361 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85"} err="failed to get container status \"ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85\": rpc error: code = NotFound desc = could not find container \"ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85\": container with ID starting with ca78f7cd2cf2fefd102b38664e0482f275d64da765683aa537025f5b08e0fa85 not found: ID does not exist" Jul 27 23:21:21.934675 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.934654 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cd8dc995c-frmhn"] Jul 27 23:21:21.938128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:21.938108 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cd8dc995c-frmhn"] Jul 27 23:21:23.643029 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:23.642992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:23.781348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:23.781318 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" path="/var/lib/kubelet/pods/560af22b-0e13-4e3b-b21f-c02c8f6011f6/volumes" Jul 27 23:21:24.288028 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:24.287995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:24.390263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:24.390232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:24.942602 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:24.942571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:24.995165 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:24.995135 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:28.948065 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:28.948027 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c23353e-81b9-4556-a3a2-33b243c63f68" containerID="cbb603943eccaed273fdc355cb2c59fb1782470ca822c2f137d7a45dd28021b2" exitCode=0 Jul 27 23:21:28.948527 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:28.948106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" event={"ID":"9c23353e-81b9-4556-a3a2-33b243c63f68","Type":"ContainerDied","Data":"cbb603943eccaed273fdc355cb2c59fb1782470ca822c2f137d7a45dd28021b2"} Jul 27 23:21:28.948593 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:28.948539 2564 scope.go:117] "RemoveContainer" containerID="cbb603943eccaed273fdc355cb2c59fb1782470ca822c2f137d7a45dd28021b2" Jul 27 23:21:29.325953 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.325918 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b46b45f6b-f5rbm" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" containerName="console" containerID="cri-o://c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4" gracePeriod=15 Jul 27 23:21:29.585304 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.585246 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:21:29.608976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.608950 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-oauth-config\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.608991 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-serving-cert\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-oauth-serving-cert\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609043 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-service-ca\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609112 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-console-config\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609323 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4msw\" (UniqueName: \"kubernetes.io/projected/58a740e4-2f35-4d48-9009-3e24d9408473-kube-api-access-l4msw\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609323 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609185 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-trusted-ca-bundle\") pod \"58a740e4-2f35-4d48-9009-3e24d9408473\" (UID: \"58a740e4-2f35-4d48-9009-3e24d9408473\") " Jul 27 23:21:29.609795 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-service-ca" (OuterVolumeSpecName: "service-ca") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:29.609927 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609793 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:29.609927 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:29.609927 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.609864 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-console-config" (OuterVolumeSpecName: "console-config") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:21:29.611976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.611952 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:21:29.611976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.611962 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a740e4-2f35-4d48-9009-3e24d9408473-kube-api-access-l4msw" (OuterVolumeSpecName: "kube-api-access-l4msw") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "kube-api-access-l4msw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:21:29.612105 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.612014 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "58a740e4-2f35-4d48-9009-3e24d9408473" (UID: "58a740e4-2f35-4d48-9009-3e24d9408473"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:21:29.710258 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710228 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-oauth-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.710258 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710256 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58a740e4-2f35-4d48-9009-3e24d9408473-console-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.710440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710268 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-oauth-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.710440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710277 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.710440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710286 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-console-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.710440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710295 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4msw\" (UniqueName: \"kubernetes.io/projected/58a740e4-2f35-4d48-9009-3e24d9408473-kube-api-access-l4msw\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.710440 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.710303 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58a740e4-2f35-4d48-9009-3e24d9408473-trusted-ca-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:21:29.857646 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.854011 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74f9f647d5-fbgwn_26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5/router/0.log" Jul 27 23:21:29.868144 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.868113 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kh2xw_75cddcee-0023-4c16-828c-2294054f6c57/serve-healthcheck-canary/0.log" Jul 27 23:21:29.953475 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.953440 2564 generic.go:356] "Generic (PLEG): container finished" podID="58a740e4-2f35-4d48-9009-3e24d9408473" containerID="c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4" exitCode=0 Jul 27 23:21:29.953970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.953493 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b46b45f6b-f5rbm" Jul 27 23:21:29.953970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.953532 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b46b45f6b-f5rbm" event={"ID":"58a740e4-2f35-4d48-9009-3e24d9408473","Type":"ContainerDied","Data":"c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4"} Jul 27 23:21:29.953970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.953617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b46b45f6b-f5rbm" event={"ID":"58a740e4-2f35-4d48-9009-3e24d9408473","Type":"ContainerDied","Data":"d70c3860109faefd4a11d6e3584d6f3e3663447a2426e81e601723bde5b77ce0"} Jul 27 23:21:29.953970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.953643 2564 scope.go:117] "RemoveContainer" containerID="c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4" Jul 27 23:21:29.955530 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.955506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-7mjm7" event={"ID":"9c23353e-81b9-4556-a3a2-33b243c63f68","Type":"ContainerStarted","Data":"f2b00981c88cd44caa0b997f840d1fb43ec87eb05dc325d14dfbe0e77310dcdf"} Jul 27 23:21:29.962387 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.962372 2564 scope.go:117] "RemoveContainer" containerID="c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4" Jul 27 23:21:29.962642 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:21:29.962623 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4\": container with ID starting with c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4 not found: ID does not exist" containerID="c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4" Jul 27 23:21:29.962691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.962651 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4"} err="failed to get container status \"c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4\": rpc error: code = NotFound desc = could not find container \"c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4\": container with ID starting with c3023e120a52e6664c9dedc64fa39cb11cf076e4edf0f134ac98cbd326ec4ea4 not found: ID does not exist" Jul 27 23:21:29.968156 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.968134 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b46b45f6b-f5rbm"] Jul 27 23:21:29.969925 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:29.969902 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b46b45f6b-f5rbm"] Jul 27 23:21:31.782030 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:31.781997 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" path="/var/lib/kubelet/pods/58a740e4-2f35-4d48-9009-3e24d9408473/volumes" Jul 27 23:21:32.637340 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:32.637303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:32.735972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:32.735939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:39.294415 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:39.294385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:39.343428 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:39.343401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:46.513404 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:46.513371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:46.614335 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:46.614306 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:46.664455 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:46.664433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:51.921909 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:51.921806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:21:52.253196 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:21:52.253158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:00.468471 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468436 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-765679cd-zq2gc"] Jul 27 23:22:00.468848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468742 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" containerName="console" Jul 27 23:22:00.468848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468758 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" containerName="console" Jul 27 23:22:00.468848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468769 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" containerName="registry" Jul 27 23:22:00.468848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468774 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" containerName="registry" Jul 27 23:22:00.468848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468785 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" containerName="console" Jul 27 23:22:00.468848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468791 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" containerName="console" Jul 27 23:22:00.469054 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468872 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a740e4-2f35-4d48-9009-3e24d9408473" containerName="console" Jul 27 23:22:00.469054 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468881 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d920ed-b61c-456f-87ad-5ba6dfd56339" containerName="console" Jul 27 23:22:00.469054 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.468886 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="560af22b-0e13-4e3b-b21f-c02c8f6011f6" containerName="registry" Jul 27 23:22:00.473326 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.473303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.483955 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.483932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765679cd-zq2gc"] Jul 27 23:22:00.525219 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.525196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:00.578000 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.577978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:00.581119 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-console-config\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.581178 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5kk\" (UniqueName: \"kubernetes.io/projected/47f89ea5-57db-443f-b1e2-0951732a66cf-kube-api-access-bx5kk\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.581178 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-oauth-serving-cert\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.581253 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-oauth-config\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.581253 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-serving-cert\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.581253 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-service-ca\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.581358 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.581270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-trusted-ca-bundle\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.671128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.671102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:00.682427 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-service-ca\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.682503 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-trusted-ca-bundle\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.682587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-console-config\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.682627 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5kk\" (UniqueName: \"kubernetes.io/projected/47f89ea5-57db-443f-b1e2-0951732a66cf-kube-api-access-bx5kk\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.682674 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-oauth-serving-cert\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.682714 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-oauth-config\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.682846 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.682801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-serving-cert\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.683326 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.683305 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-service-ca\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.683476 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.683397 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-console-config\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.683476 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.683421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-trusted-ca-bundle\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.683642 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.683552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-oauth-serving-cert\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.685336 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.685317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-oauth-config\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.685408 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.685388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-serving-cert\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.689736 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.689714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5kk\" (UniqueName: \"kubernetes.io/projected/47f89ea5-57db-443f-b1e2-0951732a66cf-kube-api-access-bx5kk\") pod \"console-765679cd-zq2gc\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.782685 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.782652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:00.910721 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:00.910579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-765679cd-zq2gc"] Jul 27 23:22:00.913442 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:22:00.913418 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f89ea5_57db_443f_b1e2_0951732a66cf.slice/crio-17b16fd5ac2ab01824a338291cb029d85f579d0c47b69179d590e97c6f1502b9 WatchSource:0}: Error finding container 17b16fd5ac2ab01824a338291cb029d85f579d0c47b69179d590e97c6f1502b9: Status 404 returned error can't find the container with id 17b16fd5ac2ab01824a338291cb029d85f579d0c47b69179d590e97c6f1502b9 Jul 27 23:22:01.087151 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:01.087063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765679cd-zq2gc" event={"ID":"47f89ea5-57db-443f-b1e2-0951732a66cf","Type":"ContainerStarted","Data":"ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0"} Jul 27 23:22:01.087151 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:01.087117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765679cd-zq2gc" event={"ID":"47f89ea5-57db-443f-b1e2-0951732a66cf","Type":"ContainerStarted","Data":"17b16fd5ac2ab01824a338291cb029d85f579d0c47b69179d590e97c6f1502b9"} Jul 27 23:22:01.102877 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:01.102831 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-765679cd-zq2gc" podStartSLOduration=1.102801111 podStartE2EDuration="1.102801111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:22:01.102284812 +0000 UTC m=+363.900249435" watchObservedRunningTime="2026-07-27 23:22:01.102801111 +0000 UTC m=+363.900765792" Jul 27 23:22:01.270213 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:01.270166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:01.370311 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:01.370222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:10.557211 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.557171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:22:10.559695 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.559670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93e62be3-2c13-473b-b298-ae8f2dde91b8-metrics-certs\") pod \"network-metrics-daemon-l6jq8\" (UID: \"93e62be3-2c13-473b-b298-ae8f2dde91b8\") " pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:22:10.579996 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.579970 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n98qx\"" Jul 27 23:22:10.587645 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.587630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6jq8" Jul 27 23:22:10.717990 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.717967 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6jq8"] Jul 27 23:22:10.720621 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:22:10.720591 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e62be3_2c13_473b_b298_ae8f2dde91b8.slice/crio-526b806c394909637c58e0d3d42edd27c2d8460261e0e4d3dda9bc6d8273a8bb WatchSource:0}: Error finding container 526b806c394909637c58e0d3d42edd27c2d8460261e0e4d3dda9bc6d8273a8bb: Status 404 returned error can't find the container with id 526b806c394909637c58e0d3d42edd27c2d8460261e0e4d3dda9bc6d8273a8bb Jul 27 23:22:10.722513 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.722498 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 23:22:10.783336 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.783301 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:10.783499 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.783357 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:10.788008 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:10.787989 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:11.131363 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:11.131330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6jq8" event={"ID":"93e62be3-2c13-473b-b298-ae8f2dde91b8","Type":"ContainerStarted","Data":"526b806c394909637c58e0d3d42edd27c2d8460261e0e4d3dda9bc6d8273a8bb"} Jul 27 23:22:11.135283 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:11.135258 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:22:11.180124 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:11.180089 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8dbd688f-2jlgc"] Jul 27 23:22:11.272210 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:11.272179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:11.373013 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:11.372981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:11.425866 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:11.425767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:12.137644 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:12.137602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6jq8" event={"ID":"93e62be3-2c13-473b-b298-ae8f2dde91b8","Type":"ContainerStarted","Data":"a5b373fcebd131fe6efc496ae5db73b5c087611166c04fa0b63f995cf0a111da"} Jul 27 23:22:12.138149 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:12.137661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6jq8" event={"ID":"93e62be3-2c13-473b-b298-ae8f2dde91b8","Type":"ContainerStarted","Data":"9a2edcfd15464902b718508984acc387952da2d69e6d62c591bc66fb3af4c05a"} Jul 27 23:22:12.152451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:12.152281 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l6jq8" podStartSLOduration=373.251467194 podStartE2EDuration="6m14.152263726s" totalImagesPullingTime="900.796532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:22:12.15175471 +0000 UTC m=+374.949719335" watchObservedRunningTime="2026-07-27 23:22:12.152263726 +0000 UTC m=+374.950228350" Jul 27 23:22:12.233778 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:12.233744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:12.287366 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:12.287336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:22:36.205229 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.205166 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f8dbd688f-2jlgc" podUID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" containerName="console" containerID="cri-o://8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565" gracePeriod=15 Jul 27 23:22:36.451099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.451077 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:22:36.582311 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582280 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-serving-cert\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-service-ca\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582345 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-oauth-config\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-trusted-ca-bundle\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582403 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-config\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-oauth-serving-cert\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582482 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg8p4\" (UniqueName: \"kubernetes.io/projected/f7caaa6f-ec23-4644-9eeb-4cd22947efab-kube-api-access-xg8p4\") pod \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\" (UID: \"f7caaa6f-ec23-4644-9eeb-4cd22947efab\") " Jul 27 23:22:36.582923 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582846 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-service-ca" (OuterVolumeSpecName: "service-ca") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:22:36.583054 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.582958 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:22:36.583109 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.583043 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:22:36.583109 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.583040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-config" (OuterVolumeSpecName: "console-config") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:22:36.584689 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.584670 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:22:36.585192 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.585167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:22:36.585311 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.585201 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7caaa6f-ec23-4644-9eeb-4cd22947efab-kube-api-access-xg8p4" (OuterVolumeSpecName: "kube-api-access-xg8p4") pod "f7caaa6f-ec23-4644-9eeb-4cd22947efab" (UID: "f7caaa6f-ec23-4644-9eeb-4cd22947efab"). InnerVolumeSpecName "kube-api-access-xg8p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:22:36.683438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683403 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-trusted-ca-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:36.683438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683433 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:36.683438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683446 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-oauth-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:36.683654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683458 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg8p4\" (UniqueName: \"kubernetes.io/projected/f7caaa6f-ec23-4644-9eeb-4cd22947efab-kube-api-access-xg8p4\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:36.683654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683472 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:36.683654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683483 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f7caaa6f-ec23-4644-9eeb-4cd22947efab-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:36.683654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:36.683494 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f7caaa6f-ec23-4644-9eeb-4cd22947efab-console-oauth-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:22:37.232479 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.232445 2564 generic.go:356] "Generic (PLEG): container finished" podID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" containerID="8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565" exitCode=0 Jul 27 23:22:37.232912 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.232490 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8dbd688f-2jlgc" Jul 27 23:22:37.232912 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.232529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8dbd688f-2jlgc" event={"ID":"f7caaa6f-ec23-4644-9eeb-4cd22947efab","Type":"ContainerDied","Data":"8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565"} Jul 27 23:22:37.232912 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.232578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8dbd688f-2jlgc" event={"ID":"f7caaa6f-ec23-4644-9eeb-4cd22947efab","Type":"ContainerDied","Data":"0f01b970ffecdcfea68ca7cf57daa2b17d47e270c846eefa97b8ad8296256f82"} Jul 27 23:22:37.232912 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.232599 2564 scope.go:117] "RemoveContainer" containerID="8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565" Jul 27 23:22:37.241556 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.241541 2564 scope.go:117] "RemoveContainer" containerID="8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565" Jul 27 23:22:37.241799 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:22:37.241781 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565\": container with ID starting with 8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565 not found: ID does not exist" containerID="8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565" Jul 27 23:22:37.241868 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.241809 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565"} err="failed to get container status \"8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565\": rpc error: code = NotFound desc = could not find container \"8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565\": container with ID starting with 8c4b1b1b232dcc3afa98f4ac98228522ff888f2c23fbb0dfff292d49a2589565 not found: ID does not exist" Jul 27 23:22:37.252138 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.252116 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8dbd688f-2jlgc"] Jul 27 23:22:37.256008 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.255985 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f8dbd688f-2jlgc"] Jul 27 23:22:37.781218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:37.781177 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" path="/var/lib/kubelet/pods/f7caaa6f-ec23-4644-9eeb-4cd22947efab/volumes" Jul 27 23:22:59.944029 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:22:59.943998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:00.001395 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:00.001370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:07.138415 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:07.138387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:07.190370 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:07.190341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:14.928662 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.928629 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c5c99cd54-tbnrc"] Jul 27 23:23:14.929060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.928929 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" containerName="console" Jul 27 23:23:14.929060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.928941 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" containerName="console" Jul 27 23:23:14.929060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.928990 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7caaa6f-ec23-4644-9eeb-4cd22947efab" containerName="console" Jul 27 23:23:14.933207 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.933184 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.942884 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.942862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5c99cd54-tbnrc"] Jul 27 23:23:14.970017 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.969997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-oauth-config\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.970126 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.970026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-serving-cert\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.970126 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.970043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-service-ca\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.970126 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.970058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-trusted-ca-bundle\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.970126 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.970080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-config\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.970281 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.970126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-oauth-serving-cert\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.970281 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.970175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vh26\" (UniqueName: \"kubernetes.io/projected/47b38bdc-82fa-4b25-8f18-3955deaf9f38-kube-api-access-7vh26\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:14.982346 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:14.982308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:15.034712 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.034683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:15.071116 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-oauth-config\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-serving-cert\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-service-ca\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-trusted-ca-bundle\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071398 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-config\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071398 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-oauth-serving-cert\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071515 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vh26\" (UniqueName: \"kubernetes.io/projected/47b38bdc-82fa-4b25-8f18-3955deaf9f38-kube-api-access-7vh26\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.071927 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.071904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-service-ca\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.072039 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.072018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-oauth-serving-cert\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.072083 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.072053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-config\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.072195 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.072179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-trusted-ca-bundle\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.074118 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.074094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-serving-cert\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.074209 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.074123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-oauth-config\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.078363 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.078342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vh26\" (UniqueName: \"kubernetes.io/projected/47b38bdc-82fa-4b25-8f18-3955deaf9f38-kube-api-access-7vh26\") pod \"console-c5c99cd54-tbnrc\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.123326 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.123300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:15.179850 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.179736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:15.242413 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.242391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:15.378441 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.378266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5c99cd54-tbnrc"] Jul 27 23:23:15.381337 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:23:15.381313 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b38bdc_82fa_4b25_8f18_3955deaf9f38.slice/crio-4f38796d558c6babaa154cce36e386485119302826a657fe54014c2aece5a158 WatchSource:0}: Error finding container 4f38796d558c6babaa154cce36e386485119302826a657fe54014c2aece5a158: Status 404 returned error can't find the container with id 4f38796d558c6babaa154cce36e386485119302826a657fe54014c2aece5a158 Jul 27 23:23:15.727388 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.727289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:15.778015 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:15.777984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:16.385337 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:16.385303 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5c99cd54-tbnrc" event={"ID":"47b38bdc-82fa-4b25-8f18-3955deaf9f38","Type":"ContainerStarted","Data":"562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b"} Jul 27 23:23:16.385763 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:16.385349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5c99cd54-tbnrc" event={"ID":"47b38bdc-82fa-4b25-8f18-3955deaf9f38","Type":"ContainerStarted","Data":"4f38796d558c6babaa154cce36e386485119302826a657fe54014c2aece5a158"} Jul 27 23:23:16.402292 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:16.402245 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c5c99cd54-tbnrc" podStartSLOduration=2.402231715 podStartE2EDuration="2.402231715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:23:16.40038505 +0000 UTC m=+439.198349675" watchObservedRunningTime="2026-07-27 23:23:16.402231715 +0000 UTC m=+439.200196340" Jul 27 23:23:19.161154 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:19.161074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:19.211653 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:19.211614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:24.301078 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:24.301048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:24.399587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:24.399560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:25.243615 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.243565 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:25.243615 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.243633 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:25.248213 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.248184 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:25.424213 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.424186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:23:25.464678 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.464648 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765679cd-zq2gc"] Jul 27 23:23:25.558258 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.558213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:25.609648 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:25.609612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:35.628181 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:35.628144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:47.659110 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:47.659070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:49.258461 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:49.258425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:49.358009 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:49.357973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:23:50.490095 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.490027 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-765679cd-zq2gc" podUID="47f89ea5-57db-443f-b1e2-0951732a66cf" containerName="console" containerID="cri-o://ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0" gracePeriod=15 Jul 27 23:23:50.736229 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.736208 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:23:50.839352 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-trusted-ca-bundle\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839352 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839367 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx5kk\" (UniqueName: \"kubernetes.io/projected/47f89ea5-57db-443f-b1e2-0951732a66cf-kube-api-access-bx5kk\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839554 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-oauth-serving-cert\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839554 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-service-ca\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839655 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-oauth-config\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839655 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839623 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-console-config\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839655 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-serving-cert\") pod \"47f89ea5-57db-443f-b1e2-0951732a66cf\" (UID: \"47f89ea5-57db-443f-b1e2-0951732a66cf\") " Jul 27 23:23:50.839807 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839769 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:23:50.839807 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.839796 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:23:50.840074 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.840031 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-trusted-ca-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:50.840074 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.840053 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-oauth-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:50.840410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.840122 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-service-ca" (OuterVolumeSpecName: "service-ca") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:23:50.840410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.840146 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-console-config" (OuterVolumeSpecName: "console-config") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:23:50.841923 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.841898 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:23:50.842008 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.841923 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f89ea5-57db-443f-b1e2-0951732a66cf-kube-api-access-bx5kk" (OuterVolumeSpecName: "kube-api-access-bx5kk") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "kube-api-access-bx5kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:23:50.842008 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.841922 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "47f89ea5-57db-443f-b1e2-0951732a66cf" (UID: "47f89ea5-57db-443f-b1e2-0951732a66cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:23:50.940629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.940596 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:50.940629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.940627 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-oauth-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:50.940629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.940638 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47f89ea5-57db-443f-b1e2-0951732a66cf-console-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:50.940785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.940648 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47f89ea5-57db-443f-b1e2-0951732a66cf-console-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:50.940785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:50.940656 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5kk\" (UniqueName: \"kubernetes.io/projected/47f89ea5-57db-443f-b1e2-0951732a66cf-kube-api-access-bx5kk\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:23:51.518779 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.518748 2564 generic.go:356] "Generic (PLEG): container finished" podID="47f89ea5-57db-443f-b1e2-0951732a66cf" containerID="ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0" exitCode=0 Jul 27 23:23:51.519166 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.518795 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-765679cd-zq2gc" Jul 27 23:23:51.519166 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.518853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765679cd-zq2gc" event={"ID":"47f89ea5-57db-443f-b1e2-0951732a66cf","Type":"ContainerDied","Data":"ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0"} Jul 27 23:23:51.519166 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.518903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-765679cd-zq2gc" event={"ID":"47f89ea5-57db-443f-b1e2-0951732a66cf","Type":"ContainerDied","Data":"17b16fd5ac2ab01824a338291cb029d85f579d0c47b69179d590e97c6f1502b9"} Jul 27 23:23:51.519166 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.518926 2564 scope.go:117] "RemoveContainer" containerID="ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0" Jul 27 23:23:51.527404 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.527391 2564 scope.go:117] "RemoveContainer" containerID="ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0" Jul 27 23:23:51.527653 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:23:51.527636 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0\": container with ID starting with ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0 not found: ID does not exist" containerID="ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0" Jul 27 23:23:51.527692 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.527661 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0"} err="failed to get container status \"ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0\": rpc error: code = NotFound desc = could not find container \"ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0\": container with ID starting with ce1d09499414e60ab8fd374505ed2f11ef6a939799a5883453445246f0f25ea0 not found: ID does not exist" Jul 27 23:23:51.537950 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.537930 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-765679cd-zq2gc"] Jul 27 23:23:51.541197 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.541180 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-765679cd-zq2gc"] Jul 27 23:23:51.782378 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:51.782307 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47f89ea5-57db-443f-b1e2-0951732a66cf" path="/var/lib/kubelet/pods/47f89ea5-57db-443f-b1e2-0951732a66cf/volumes" Jul 27 23:23:57.245164 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.245135 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn"] Jul 27 23:23:57.245604 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.245547 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47f89ea5-57db-443f-b1e2-0951732a66cf" containerName="console" Jul 27 23:23:57.245604 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.245564 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="47f89ea5-57db-443f-b1e2-0951732a66cf" containerName="console" Jul 27 23:23:57.245718 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.245640 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="47f89ea5-57db-443f-b1e2-0951732a66cf" containerName="console" Jul 27 23:23:57.249956 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.249937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.252205 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.252182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-86skg\"" Jul 27 23:23:57.252205 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.252200 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 23:23:57.253142 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.253123 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 23:23:57.255077 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.255058 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn"] Jul 27 23:23:57.383101 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.383068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28xx\" (UniqueName: \"kubernetes.io/projected/15e4cb8d-17c3-4137-88ae-fe617562fbe2-kube-api-access-w28xx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.383101 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.383109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.383261 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.383192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.483987 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.483965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.484120 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.484008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28xx\" (UniqueName: \"kubernetes.io/projected/15e4cb8d-17c3-4137-88ae-fe617562fbe2-kube-api-access-w28xx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.484120 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.484029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.484337 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.484320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.484390 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.484336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.491282 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.491264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28xx\" (UniqueName: \"kubernetes.io/projected/15e4cb8d-17c3-4137-88ae-fe617562fbe2-kube-api-access-w28xx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.559920 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.559903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:23:57.685848 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:57.685803 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn"] Jul 27 23:23:57.688296 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:23:57.688271 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e4cb8d_17c3_4137_88ae_fe617562fbe2.slice/crio-04223ba72737ed3fff4792ef2aebf9a1a108d47eb31832bcab7ddce4124c45c0 WatchSource:0}: Error finding container 04223ba72737ed3fff4792ef2aebf9a1a108d47eb31832bcab7ddce4124c45c0: Status 404 returned error can't find the container with id 04223ba72737ed3fff4792ef2aebf9a1a108d47eb31832bcab7ddce4124c45c0 Jul 27 23:23:58.548210 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:23:58.548168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" event={"ID":"15e4cb8d-17c3-4137-88ae-fe617562fbe2","Type":"ContainerStarted","Data":"04223ba72737ed3fff4792ef2aebf9a1a108d47eb31832bcab7ddce4124c45c0"} Jul 27 23:24:04.573184 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:04.573155 2564 generic.go:356] "Generic (PLEG): container finished" podID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerID="8529afc7142a71b9186aa4b36cc8242822c6b7d874ea91f17998c3eb3f667725" exitCode=0 Jul 27 23:24:04.573533 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:04.573233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" event={"ID":"15e4cb8d-17c3-4137-88ae-fe617562fbe2","Type":"ContainerDied","Data":"8529afc7142a71b9186aa4b36cc8242822c6b7d874ea91f17998c3eb3f667725"} Jul 27 23:24:07.586679 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:07.586645 2564 generic.go:356] "Generic (PLEG): container finished" podID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerID="5dfe432c97d946c3bd72b71495fa30ca4e4228ab3cfe8d2f292308609fdc200d" exitCode=0 Jul 27 23:24:07.587067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:07.586694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" event={"ID":"15e4cb8d-17c3-4137-88ae-fe617562fbe2","Type":"ContainerDied","Data":"5dfe432c97d946c3bd72b71495fa30ca4e4228ab3cfe8d2f292308609fdc200d"} Jul 27 23:24:15.620736 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:15.620698 2564 generic.go:356] "Generic (PLEG): container finished" podID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerID="6379dd3f277e3d8603300778ef2cc5d96dedc0e45c8ba104bdf929e009c25261" exitCode=0 Jul 27 23:24:15.621163 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:15.620781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" event={"ID":"15e4cb8d-17c3-4137-88ae-fe617562fbe2","Type":"ContainerDied","Data":"6379dd3f277e3d8603300778ef2cc5d96dedc0e45c8ba104bdf929e009c25261"} Jul 27 23:24:16.746325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.746303 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:24:16.846678 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.846652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-util\") pod \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " Jul 27 23:24:16.846836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.846697 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w28xx\" (UniqueName: \"kubernetes.io/projected/15e4cb8d-17c3-4137-88ae-fe617562fbe2-kube-api-access-w28xx\") pod \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " Jul 27 23:24:16.846836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.846741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-bundle\") pod \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\" (UID: \"15e4cb8d-17c3-4137-88ae-fe617562fbe2\") " Jul 27 23:24:16.847447 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.847371 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-bundle" (OuterVolumeSpecName: "bundle") pod "15e4cb8d-17c3-4137-88ae-fe617562fbe2" (UID: "15e4cb8d-17c3-4137-88ae-fe617562fbe2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:24:16.849007 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.848980 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e4cb8d-17c3-4137-88ae-fe617562fbe2-kube-api-access-w28xx" (OuterVolumeSpecName: "kube-api-access-w28xx") pod "15e4cb8d-17c3-4137-88ae-fe617562fbe2" (UID: "15e4cb8d-17c3-4137-88ae-fe617562fbe2"). InnerVolumeSpecName "kube-api-access-w28xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:24:16.850708 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.850691 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-util" (OuterVolumeSpecName: "util") pod "15e4cb8d-17c3-4137-88ae-fe617562fbe2" (UID: "15e4cb8d-17c3-4137-88ae-fe617562fbe2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:24:16.947614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.947534 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:24:16.947614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.947570 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e4cb8d-17c3-4137-88ae-fe617562fbe2-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:24:16.947614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:16.947578 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28xx\" (UniqueName: \"kubernetes.io/projected/15e4cb8d-17c3-4137-88ae-fe617562fbe2-kube-api-access-w28xx\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:24:17.629845 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:17.629804 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" Jul 27 23:24:17.629845 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:17.629805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bskgn" event={"ID":"15e4cb8d-17c3-4137-88ae-fe617562fbe2","Type":"ContainerDied","Data":"04223ba72737ed3fff4792ef2aebf9a1a108d47eb31832bcab7ddce4124c45c0"} Jul 27 23:24:17.630041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:17.629866 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04223ba72737ed3fff4792ef2aebf9a1a108d47eb31832bcab7ddce4124c45c0" Jul 27 23:24:23.781226 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:23.781196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:25.356301 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.356265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:25.558041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558007 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6"] Jul 27 23:24:25.558289 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558277 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="pull" Jul 27 23:24:25.558330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558291 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="pull" Jul 27 23:24:25.558330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558303 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="util" Jul 27 23:24:25.558330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558308 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="util" Jul 27 23:24:25.558330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558318 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="extract" Jul 27 23:24:25.558330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558324 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="extract" Jul 27 23:24:25.558471 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.558373 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e4cb8d-17c3-4137-88ae-fe617562fbe2" containerName="extract" Jul 27 23:24:25.560243 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.560227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.562643 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.562620 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 27 23:24:25.562767 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.562702 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:24:25.562846 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.562772 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xf8cf\"" Jul 27 23:24:25.571246 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.571226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6"] Jul 27 23:24:25.606888 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.606836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cqm\" (UniqueName: \"kubernetes.io/projected/31df3ed9-9623-4115-b8f1-50daa2adf96e-kube-api-access-p4cqm\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvbt6\" (UID: \"31df3ed9-9623-4115-b8f1-50daa2adf96e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.606888 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.606880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31df3ed9-9623-4115-b8f1-50daa2adf96e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvbt6\" (UID: \"31df3ed9-9623-4115-b8f1-50daa2adf96e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.707976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.707950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cqm\" (UniqueName: \"kubernetes.io/projected/31df3ed9-9623-4115-b8f1-50daa2adf96e-kube-api-access-p4cqm\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvbt6\" (UID: \"31df3ed9-9623-4115-b8f1-50daa2adf96e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.708067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.707983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31df3ed9-9623-4115-b8f1-50daa2adf96e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvbt6\" (UID: \"31df3ed9-9623-4115-b8f1-50daa2adf96e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.708289 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.708274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31df3ed9-9623-4115-b8f1-50daa2adf96e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvbt6\" (UID: \"31df3ed9-9623-4115-b8f1-50daa2adf96e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.722579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.722559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cqm\" (UniqueName: \"kubernetes.io/projected/31df3ed9-9623-4115-b8f1-50daa2adf96e-kube-api-access-p4cqm\") pod \"cert-manager-operator-controller-manager-55c444bf86-mvbt6\" (UID: \"31df3ed9-9623-4115-b8f1-50daa2adf96e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:25.869541 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:25.869471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" Jul 27 23:24:26.015588 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:26.015564 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6"] Jul 27 23:24:26.018413 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:24:26.018382 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31df3ed9_9623_4115_b8f1_50daa2adf96e.slice/crio-8fb9d25220621246c556d7e024bc54615c374726371886de83571eeeecb055a2 WatchSource:0}: Error finding container 8fb9d25220621246c556d7e024bc54615c374726371886de83571eeeecb055a2: Status 404 returned error can't find the container with id 8fb9d25220621246c556d7e024bc54615c374726371886de83571eeeecb055a2 Jul 27 23:24:26.663945 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:26.663910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" event={"ID":"31df3ed9-9623-4115-b8f1-50daa2adf96e","Type":"ContainerStarted","Data":"8fb9d25220621246c556d7e024bc54615c374726371886de83571eeeecb055a2"} Jul 27 23:24:27.181836 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:27.181780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:27.281884 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:27.281832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:28.574585 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.574551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:28.673426 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.673398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:28.674672 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.674650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" event={"ID":"31df3ed9-9623-4115-b8f1-50daa2adf96e","Type":"ContainerStarted","Data":"ec338ac5f7fe337468cb901bd4479531f3f99e48121cc76795f9d91200a59f2e"} Jul 27 23:24:28.683626 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.683602 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c5lxr"] Jul 27 23:24:28.685793 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.685779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:28.688016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.687999 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 23:24:28.688016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.688012 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 23:24:28.694044 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.694023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c5lxr"] Jul 27 23:24:28.701402 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.700603 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-mvbt6" podStartSLOduration=1.72510037 podStartE2EDuration="3.700587895s" totalImagesPullingTime="1.975487525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:24:28.699051054 +0000 UTC m=+511.497015678" watchObservedRunningTime="2026-07-27 23:24:28.700587895 +0000 UTC m=+511.498552519" Jul 27 23:24:28.732271 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.732247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lwq\" (UniqueName: \"kubernetes.io/projected/f30e7775-5f1c-4875-b4b1-dec577f49fdc-kube-api-access-x7lwq\") pod \"cert-manager-webhook-6486ffd997-c5lxr\" (UID: \"f30e7775-5f1c-4875-b4b1-dec577f49fdc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:28.732382 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.732299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f30e7775-5f1c-4875-b4b1-dec577f49fdc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c5lxr\" (UID: \"f30e7775-5f1c-4875-b4b1-dec577f49fdc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:28.832862 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.832776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lwq\" (UniqueName: \"kubernetes.io/projected/f30e7775-5f1c-4875-b4b1-dec577f49fdc-kube-api-access-x7lwq\") pod \"cert-manager-webhook-6486ffd997-c5lxr\" (UID: \"f30e7775-5f1c-4875-b4b1-dec577f49fdc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:28.832969 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.832869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f30e7775-5f1c-4875-b4b1-dec577f49fdc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c5lxr\" (UID: \"f30e7775-5f1c-4875-b4b1-dec577f49fdc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:28.841024 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.840995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lwq\" (UniqueName: \"kubernetes.io/projected/f30e7775-5f1c-4875-b4b1-dec577f49fdc-kube-api-access-x7lwq\") pod \"cert-manager-webhook-6486ffd997-c5lxr\" (UID: \"f30e7775-5f1c-4875-b4b1-dec577f49fdc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:28.841024 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:28.841004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f30e7775-5f1c-4875-b4b1-dec577f49fdc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c5lxr\" (UID: \"f30e7775-5f1c-4875-b4b1-dec577f49fdc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:29.002062 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:29.002034 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:29.110865 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:29.110810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:29.150364 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:29.150345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c5lxr"] Jul 27 23:24:29.153836 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:24:29.153786 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf30e7775_5f1c_4875_b4b1_dec577f49fdc.slice/crio-e8809db014611e312b5e2291492100e665eb23f802545cac04a4738645757244 WatchSource:0}: Error finding container e8809db014611e312b5e2291492100e665eb23f802545cac04a4738645757244: Status 404 returned error can't find the container with id e8809db014611e312b5e2291492100e665eb23f802545cac04a4738645757244 Jul 27 23:24:29.209077 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:29.209045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:29.680102 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:29.680069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" event={"ID":"f30e7775-5f1c-4875-b4b1-dec577f49fdc","Type":"ContainerStarted","Data":"e8809db014611e312b5e2291492100e665eb23f802545cac04a4738645757244"} Jul 27 23:24:30.403634 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.403601 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96"] Jul 27 23:24:30.406493 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.406478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.408596 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.408577 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 23:24:30.408698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.408582 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-86skg\"" Jul 27 23:24:30.409448 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.409430 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 23:24:30.414652 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.414629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96"] Jul 27 23:24:30.444364 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.444344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pv7k\" (UniqueName: \"kubernetes.io/projected/37ab712b-a672-431f-9c3f-4c5f6695f2ce-kube-api-access-7pv7k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.444451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.444381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.444451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.444417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.545108 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.545076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pv7k\" (UniqueName: \"kubernetes.io/projected/37ab712b-a672-431f-9c3f-4c5f6695f2ce-kube-api-access-7pv7k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.545246 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.545143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.545246 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.545202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.545558 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.545532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.545558 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.545547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.553708 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.553686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pv7k\" (UniqueName: \"kubernetes.io/projected/37ab712b-a672-431f-9c3f-4c5f6695f2ce-kube-api-access-7pv7k\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.577812 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.577780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:30.716861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.716769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:30.863301 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.863279 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96"] Jul 27 23:24:30.866728 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:24:30.866689 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ab712b_a672_431f_9c3f_4c5f6695f2ce.slice/crio-00e5f731c668cf891a0d08334c4ee9789016b9de135af55e7e16c7d0f4e0c184 WatchSource:0}: Error finding container 00e5f731c668cf891a0d08334c4ee9789016b9de135af55e7e16c7d0f4e0c184: Status 404 returned error can't find the container with id 00e5f731c668cf891a0d08334c4ee9789016b9de135af55e7e16c7d0f4e0c184 Jul 27 23:24:30.967065 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:30.966953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:31.692475 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:31.692435 2564 generic.go:356] "Generic (PLEG): container finished" podID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerID="c521ab83355239cef1caa039bc0c800b60514f2d642802e817530775604e9410" exitCode=0 Jul 27 23:24:31.692653 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:31.692528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" event={"ID":"37ab712b-a672-431f-9c3f-4c5f6695f2ce","Type":"ContainerDied","Data":"c521ab83355239cef1caa039bc0c800b60514f2d642802e817530775604e9410"} Jul 27 23:24:31.692653 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:31.692596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" event={"ID":"37ab712b-a672-431f-9c3f-4c5f6695f2ce","Type":"ContainerStarted","Data":"00e5f731c668cf891a0d08334c4ee9789016b9de135af55e7e16c7d0f4e0c184"} Jul 27 23:24:32.699755 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:32.699683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" event={"ID":"f30e7775-5f1c-4875-b4b1-dec577f49fdc","Type":"ContainerStarted","Data":"954edb7a0753c6a7570f45b6b0f7b5b96db9631699a6359ba903c470563cdad4"} Jul 27 23:24:32.699755 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:32.699754 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:32.715138 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:32.715085 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" podStartSLOduration=1.404476617 podStartE2EDuration="4.715071029s" totalImagesPullingTime="3.310594412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:24:32.712157433 +0000 UTC m=+515.510122058" watchObservedRunningTime="2026-07-27 23:24:32.715071029 +0000 UTC m=+515.513035651" Jul 27 23:24:34.176528 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:34.176496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:34.276807 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:34.276777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:34.710872 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:34.710767 2564 generic.go:356] "Generic (PLEG): container finished" podID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerID="486d06e1bd861177d7c3cc94d448968f4601ae6e683eed966051682cbf2207f6" exitCode=0 Jul 27 23:24:34.710872 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:34.710854 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" event={"ID":"37ab712b-a672-431f-9c3f-4c5f6695f2ce","Type":"ContainerDied","Data":"486d06e1bd861177d7c3cc94d448968f4601ae6e683eed966051682cbf2207f6"} Jul 27 23:24:35.716764 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:35.716729 2564 generic.go:356] "Generic (PLEG): container finished" podID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerID="b66e2dfd628a7bd3ba34dcde0c1571e76289e25c0d800500c26d01c2385aa02f" exitCode=0 Jul 27 23:24:35.717129 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:35.716832 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" event={"ID":"37ab712b-a672-431f-9c3f-4c5f6695f2ce","Type":"ContainerDied","Data":"b66e2dfd628a7bd3ba34dcde0c1571e76289e25c0d800500c26d01c2385aa02f"} Jul 27 23:24:36.098699 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.098657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:36.199973 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.199944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:36.846349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.846325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:36.897190 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.897169 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-util\") pod \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " Jul 27 23:24:36.897315 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.897254 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pv7k\" (UniqueName: \"kubernetes.io/projected/37ab712b-a672-431f-9c3f-4c5f6695f2ce-kube-api-access-7pv7k\") pod \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " Jul 27 23:24:36.897315 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.897286 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-bundle\") pod \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\" (UID: \"37ab712b-a672-431f-9c3f-4c5f6695f2ce\") " Jul 27 23:24:36.897665 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.897641 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-bundle" (OuterVolumeSpecName: "bundle") pod "37ab712b-a672-431f-9c3f-4c5f6695f2ce" (UID: "37ab712b-a672-431f-9c3f-4c5f6695f2ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:24:36.899489 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.899466 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ab712b-a672-431f-9c3f-4c5f6695f2ce-kube-api-access-7pv7k" (OuterVolumeSpecName: "kube-api-access-7pv7k") pod "37ab712b-a672-431f-9c3f-4c5f6695f2ce" (UID: "37ab712b-a672-431f-9c3f-4c5f6695f2ce"). InnerVolumeSpecName "kube-api-access-7pv7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:24:36.901716 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.901689 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-util" (OuterVolumeSpecName: "util") pod "37ab712b-a672-431f-9c3f-4c5f6695f2ce" (UID: "37ab712b-a672-431f-9c3f-4c5f6695f2ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:24:36.998212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.998153 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pv7k\" (UniqueName: \"kubernetes.io/projected/37ab712b-a672-431f-9c3f-4c5f6695f2ce-kube-api-access-7pv7k\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:24:36.998212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.998178 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:24:36.998212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:36.998192 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37ab712b-a672-431f-9c3f-4c5f6695f2ce-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:24:37.729388 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:37.729352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" event={"ID":"37ab712b-a672-431f-9c3f-4c5f6695f2ce","Type":"ContainerDied","Data":"00e5f731c668cf891a0d08334c4ee9789016b9de135af55e7e16c7d0f4e0c184"} Jul 27 23:24:37.729388 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:37.729380 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbgp96" Jul 27 23:24:37.729583 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:37.729396 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00e5f731c668cf891a0d08334c4ee9789016b9de135af55e7e16c7d0f4e0c184" Jul 27 23:24:38.705981 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:38.705952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c5lxr" Jul 27 23:24:39.778122 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:39.778085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:45.797367 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:45.797321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:45.897024 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:45.896985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:47.821016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.820985 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w"] Jul 27 23:24:47.821361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821298 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="pull" Jul 27 23:24:47.821361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821310 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="pull" Jul 27 23:24:47.821361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821327 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="extract" Jul 27 23:24:47.821361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821336 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="extract" Jul 27 23:24:47.821361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821345 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="util" Jul 27 23:24:47.821361 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821350 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="util" Jul 27 23:24:47.821537 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.821403 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ab712b-a672-431f-9c3f-4c5f6695f2ce" containerName="extract" Jul 27 23:24:47.862946 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.862924 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w"] Jul 27 23:24:47.863081 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.863002 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:47.865484 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.865453 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 27 23:24:47.865484 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.865460 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kbdgn\"" Jul 27 23:24:47.866386 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.866367 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 27 23:24:47.975298 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.975271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0efdf39-72f8-4309-b24d-81f6657a451d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jl79w\" (UID: \"c0efdf39-72f8-4309-b24d-81f6657a451d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:47.975424 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:47.975322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kjw\" (UniqueName: \"kubernetes.io/projected/c0efdf39-72f8-4309-b24d-81f6657a451d-kube-api-access-h8kjw\") pod \"openshift-lws-operator-fd8ccff4c-jl79w\" (UID: \"c0efdf39-72f8-4309-b24d-81f6657a451d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:48.076476 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.076408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0efdf39-72f8-4309-b24d-81f6657a451d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jl79w\" (UID: \"c0efdf39-72f8-4309-b24d-81f6657a451d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:48.076476 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.076454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kjw\" (UniqueName: \"kubernetes.io/projected/c0efdf39-72f8-4309-b24d-81f6657a451d-kube-api-access-h8kjw\") pod \"openshift-lws-operator-fd8ccff4c-jl79w\" (UID: \"c0efdf39-72f8-4309-b24d-81f6657a451d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:48.076804 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.076785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0efdf39-72f8-4309-b24d-81f6657a451d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jl79w\" (UID: \"c0efdf39-72f8-4309-b24d-81f6657a451d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:48.083967 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.083946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kjw\" (UniqueName: \"kubernetes.io/projected/c0efdf39-72f8-4309-b24d-81f6657a451d-kube-api-access-h8kjw\") pod \"openshift-lws-operator-fd8ccff4c-jl79w\" (UID: \"c0efdf39-72f8-4309-b24d-81f6657a451d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:48.171803 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.171774 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" Jul 27 23:24:48.310896 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.310867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w"] Jul 27 23:24:48.312981 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:24:48.312953 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0efdf39_72f8_4309_b24d_81f6657a451d.slice/crio-ea919baadcf140f0abbbaaf8efc185666e2a0648a08251a42c2c4c3d4c5244ac WatchSource:0}: Error finding container ea919baadcf140f0abbbaaf8efc185666e2a0648a08251a42c2c4c3d4c5244ac: Status 404 returned error can't find the container with id ea919baadcf140f0abbbaaf8efc185666e2a0648a08251a42c2c4c3d4c5244ac Jul 27 23:24:48.776061 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:48.775991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" event={"ID":"c0efdf39-72f8-4309-b24d-81f6657a451d","Type":"ContainerStarted","Data":"ea919baadcf140f0abbbaaf8efc185666e2a0648a08251a42c2c4c3d4c5244ac"} Jul 27 23:24:50.788541 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:50.788510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" event={"ID":"c0efdf39-72f8-4309-b24d-81f6657a451d","Type":"ContainerStarted","Data":"bdfbe04315f6c7b0a40a2f512886550c88cf23099d1f932dded6aa984383bec6"} Jul 27 23:24:50.805028 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:50.804976 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jl79w" podStartSLOduration=1.774932534 podStartE2EDuration="3.804957581s" totalImagesPullingTime="2.030025047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:24:50.802388703 +0000 UTC m=+533.600353325" watchObservedRunningTime="2026-07-27 23:24:50.804957581 +0000 UTC m=+533.602922207" Jul 27 23:24:51.526212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:51.526175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:51.626515 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:51.626471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:53.689272 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:53.689239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:55.548246 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.548207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:55.603157 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.603129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:24:55.666883 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.666855 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg"] Jul 27 23:24:55.669221 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.669206 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.671276 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.671256 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 27 23:24:55.671378 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.671294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-86skg\"" Jul 27 23:24:55.672027 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.672006 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 27 23:24:55.679407 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.679387 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg"] Jul 27 23:24:55.739298 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.739274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxwd\" (UniqueName: \"kubernetes.io/projected/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-kube-api-access-fhxwd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.739387 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.739323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.739430 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.739404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.839760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.839701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.839760 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.839749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.839970 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.839924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxwd\" (UniqueName: \"kubernetes.io/projected/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-kube-api-access-fhxwd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.840129 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.840109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.840176 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.840125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.853067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.853039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxwd\" (UniqueName: \"kubernetes.io/projected/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-kube-api-access-fhxwd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:55.978136 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:55.978103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:24:56.101437 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:56.101414 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg"] Jul 27 23:24:56.103932 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:24:56.103906 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c8b9fd_c0a1_4c39_bb55_aa28af8d554a.slice/crio-706cc504344eae83a45e815a55bbebd1580fd95420c6b07a37c70e194a430fd3 WatchSource:0}: Error finding container 706cc504344eae83a45e815a55bbebd1580fd95420c6b07a37c70e194a430fd3: Status 404 returned error can't find the container with id 706cc504344eae83a45e815a55bbebd1580fd95420c6b07a37c70e194a430fd3 Jul 27 23:24:56.814791 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:56.814758 2564 generic.go:356] "Generic (PLEG): container finished" podID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerID="ef0ba333370b33d126c526c5f719330672134c8ed0247f04dd75e71dc6c26604" exitCode=0 Jul 27 23:24:56.815211 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:56.814849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" event={"ID":"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a","Type":"ContainerDied","Data":"ef0ba333370b33d126c526c5f719330672134c8ed0247f04dd75e71dc6c26604"} Jul 27 23:24:56.815211 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:56.814898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" event={"ID":"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a","Type":"ContainerStarted","Data":"706cc504344eae83a45e815a55bbebd1580fd95420c6b07a37c70e194a430fd3"} Jul 27 23:24:57.821257 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:57.821222 2564 generic.go:356] "Generic (PLEG): container finished" podID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerID="4df027f739983a520d331f5c502129788a5bfc8e449abf04140a294ca1b42bb2" exitCode=0 Jul 27 23:24:57.821604 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:57.821265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" event={"ID":"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a","Type":"ContainerDied","Data":"4df027f739983a520d331f5c502129788a5bfc8e449abf04140a294ca1b42bb2"} Jul 27 23:24:58.827600 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:58.827570 2564 generic.go:356] "Generic (PLEG): container finished" podID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerID="cd52cf2ccf3b221213dfdae0fa9ab7bccbe6091db82f3d5d3023d32a888b08bb" exitCode=0 Jul 27 23:24:58.827984 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:58.827658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" event={"ID":"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a","Type":"ContainerDied","Data":"cd52cf2ccf3b221213dfdae0fa9ab7bccbe6091db82f3d5d3023d32a888b08bb"} Jul 27 23:24:59.959870 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:24:59.959849 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:25:00.077414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.077383 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-util\") pod \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " Jul 27 23:25:00.077414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.077416 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-bundle\") pod \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " Jul 27 23:25:00.077625 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.077507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhxwd\" (UniqueName: \"kubernetes.io/projected/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-kube-api-access-fhxwd\") pod \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\" (UID: \"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a\") " Jul 27 23:25:00.078178 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.078155 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-bundle" (OuterVolumeSpecName: "bundle") pod "84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" (UID: "84c8b9fd-c0a1-4c39-bb55-aa28af8d554a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:25:00.079739 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.079716 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-kube-api-access-fhxwd" (OuterVolumeSpecName: "kube-api-access-fhxwd") pod "84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" (UID: "84c8b9fd-c0a1-4c39-bb55-aa28af8d554a"). InnerVolumeSpecName "kube-api-access-fhxwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:25:00.083026 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.082972 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-util" (OuterVolumeSpecName: "util") pod "84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" (UID: "84c8b9fd-c0a1-4c39-bb55-aa28af8d554a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:25:00.178266 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.178242 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxwd\" (UniqueName: \"kubernetes.io/projected/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-kube-api-access-fhxwd\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:25:00.178266 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.178262 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:25:00.178266 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.178272 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84c8b9fd-c0a1-4c39-bb55-aa28af8d554a-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:25:00.837447 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.837414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" event={"ID":"84c8b9fd-c0a1-4c39-bb55-aa28af8d554a","Type":"ContainerDied","Data":"706cc504344eae83a45e815a55bbebd1580fd95420c6b07a37c70e194a430fd3"} Jul 27 23:25:00.837614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.837452 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbgfzg" Jul 27 23:25:00.837614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:00.837459 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706cc504344eae83a45e815a55bbebd1580fd95420c6b07a37c70e194a430fd3" Jul 27 23:25:07.958153 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:07.958125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:08.041827 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.041792 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5"] Jul 27 23:25:08.042124 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042112 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="util" Jul 27 23:25:08.042174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042126 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="util" Jul 27 23:25:08.042174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042145 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="extract" Jul 27 23:25:08.042174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042150 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="extract" Jul 27 23:25:08.042174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042159 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="pull" Jul 27 23:25:08.042174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042166 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="pull" Jul 27 23:25:08.042314 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.042213 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c8b9fd-c0a1-4c39-bb55-aa28af8d554a" containerName="extract" Jul 27 23:25:08.044074 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.044060 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.047239 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.047217 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 27 23:25:08.047239 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.047237 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 27 23:25:08.047452 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.047217 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 27 23:25:08.047452 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.047216 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8t6lg\"" Jul 27 23:25:08.054623 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.054600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5"] Jul 27 23:25:08.058012 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.057992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:08.141443 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.141416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmnd\" (UniqueName: \"kubernetes.io/projected/f2e5233f-37fe-4133-b83a-6b2679ee8358-kube-api-access-nrmnd\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.141580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.141451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2e5233f-37fe-4133-b83a-6b2679ee8358-metrics-cert\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.141580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.141479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e5233f-37fe-4133-b83a-6b2679ee8358-cert\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.141683 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.141584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2e5233f-37fe-4133-b83a-6b2679ee8358-manager-config\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.242984 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.242904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmnd\" (UniqueName: \"kubernetes.io/projected/f2e5233f-37fe-4133-b83a-6b2679ee8358-kube-api-access-nrmnd\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.242984 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.242951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2e5233f-37fe-4133-b83a-6b2679ee8358-metrics-cert\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.242984 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.242980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e5233f-37fe-4133-b83a-6b2679ee8358-cert\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.243212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.243023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2e5233f-37fe-4133-b83a-6b2679ee8358-manager-config\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.243653 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.243633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2e5233f-37fe-4133-b83a-6b2679ee8358-manager-config\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.245475 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.245450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2e5233f-37fe-4133-b83a-6b2679ee8358-metrics-cert\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.245592 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.245576 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e5233f-37fe-4133-b83a-6b2679ee8358-cert\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.263254 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.263230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmnd\" (UniqueName: \"kubernetes.io/projected/f2e5233f-37fe-4133-b83a-6b2679ee8358-kube-api-access-nrmnd\") pod \"lws-controller-manager-6589476d7c-jqxb5\" (UID: \"f2e5233f-37fe-4133-b83a-6b2679ee8358\") " pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.353350 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.353314 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:08.483691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.483662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5"] Jul 27 23:25:08.486594 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:25:08.486571 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e5233f_37fe_4133_b83a_6b2679ee8358.slice/crio-8870f9a4729e44f7e9a7865b3cf4bf038b9e510c644cdc5174b521c18abbb353 WatchSource:0}: Error finding container 8870f9a4729e44f7e9a7865b3cf4bf038b9e510c644cdc5174b521c18abbb353: Status 404 returned error can't find the container with id 8870f9a4729e44f7e9a7865b3cf4bf038b9e510c644cdc5174b521c18abbb353 Jul 27 23:25:08.630614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.630581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:08.874892 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.874852 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" event={"ID":"f2e5233f-37fe-4133-b83a-6b2679ee8358","Type":"ContainerStarted","Data":"8870f9a4729e44f7e9a7865b3cf4bf038b9e510c644cdc5174b521c18abbb353"} Jul 27 23:25:08.936469 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:08.936390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.041397 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.041366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.075004 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.074971 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v"] Jul 27 23:25:09.078494 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.078472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.081607 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.081588 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 23:25:09.082429 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.082414 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 23:25:09.086840 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.083106 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 23:25:09.086840 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.083140 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ts86f\"" Jul 27 23:25:09.086840 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.083205 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 23:25:09.101644 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.101619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v"] Jul 27 23:25:09.149355 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.149329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.149446 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.149357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.149446 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.149387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.149549 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.149456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62cjg\" (UniqueName: \"kubernetes.io/projected/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-kube-api-access-62cjg\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.149549 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.149495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.229271 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.229190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.250321 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.250295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62cjg\" (UniqueName: \"kubernetes.io/projected/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-kube-api-access-62cjg\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.250451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.250346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.250451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.250378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.250451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.250402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.250451 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.250445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.250916 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.250790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-manifests\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.253549 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.253525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-webhook-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.253743 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.253720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.254143 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.254119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-tmp\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.258208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.258188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62cjg\" (UniqueName: \"kubernetes.io/projected/12d0bafa-ce6a-4855-8e0c-21fd85b18ed8-kube-api-access-62cjg\") pod \"opendatahub-operator-controller-manager-9cf867746-xjg9v\" (UID: \"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8\") " pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.329155 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.329130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.391190 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.391162 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:09.428645 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.428616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.495143 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.495111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.541020 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.540898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v"] Jul 27 23:25:09.544109 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:25:09.544086 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d0bafa_ce6a_4855_8e0c_21fd85b18ed8.slice/crio-e4842a8fa5cd76fc8b6f4d4655757586ad5c6bcdf18d283d0c6ca64cf7452b42 WatchSource:0}: Error finding container e4842a8fa5cd76fc8b6f4d4655757586ad5c6bcdf18d283d0c6ca64cf7452b42: Status 404 returned error can't find the container with id e4842a8fa5cd76fc8b6f4d4655757586ad5c6bcdf18d283d0c6ca64cf7452b42 Jul 27 23:25:09.567568 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.567543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:09.881734 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:09.881696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" event={"ID":"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8","Type":"ContainerStarted","Data":"e4842a8fa5cd76fc8b6f4d4655757586ad5c6bcdf18d283d0c6ca64cf7452b42"} Jul 27 23:25:10.890199 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:10.890159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" event={"ID":"f2e5233f-37fe-4133-b83a-6b2679ee8358","Type":"ContainerStarted","Data":"51c9bfd8b68e3c68e5df5ae3bfd85a975d082d62ef90703b5c1d341f1a18cbd6"} Jul 27 23:25:10.890608 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:10.890528 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:10.908655 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:10.908224 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" podStartSLOduration=1.38981749 podStartE2EDuration="2.908208282s" totalImagesPullingTime="1.518390792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:10.908055345 +0000 UTC m=+553.706019968" watchObservedRunningTime="2026-07-27 23:25:10.908208282 +0000 UTC m=+553.706172905" Jul 27 23:25:12.902091 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:12.902059 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" event={"ID":"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8","Type":"ContainerStarted","Data":"64052ba7b410854dcefc9e083e22a35ba629e9ce69efda5da23b53ab14360db8"} Jul 27 23:25:13.908510 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:13.908423 2564 generic.go:356] "Generic (PLEG): container finished" podID="12d0bafa-ce6a-4855-8e0c-21fd85b18ed8" containerID="64052ba7b410854dcefc9e083e22a35ba629e9ce69efda5da23b53ab14360db8" exitCode=0 Jul 27 23:25:13.908510 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:13.908485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" event={"ID":"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8","Type":"ContainerDied","Data":"64052ba7b410854dcefc9e083e22a35ba629e9ce69efda5da23b53ab14360db8"} Jul 27 23:25:14.916338 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:14.916269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" event={"ID":"12d0bafa-ce6a-4855-8e0c-21fd85b18ed8","Type":"ContainerStarted","Data":"4140889c3ade98e6dcf93100ce3a3da81d386213f62696e8dd1dedbc1a2e9d95"} Jul 27 23:25:14.916738 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:14.916421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:14.937481 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:14.937425 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" podStartSLOduration=1.592585051 podStartE2EDuration="5.937412278s" totalImagesPullingTime="3.298057103s" totalInitContainerRuntime="1.046770124s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:14.935874229 +0000 UTC m=+557.733838853" watchObservedRunningTime="2026-07-27 23:25:14.937412278 +0000 UTC m=+557.735376901" Jul 27 23:25:21.898313 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:21.898286 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6589476d7c-jqxb5" Jul 27 23:25:23.171701 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:23.171664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:23.218483 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:23.218457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:25.923437 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:25.923404 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9cf867746-xjg9v" Jul 27 23:25:28.053095 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.053063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:28.161149 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.161115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:28.374762 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.374667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:28.502715 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.502686 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5946b997db-jp7t2"] Jul 27 23:25:28.509230 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.509210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.511585 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.511563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 23:25:28.511835 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.511801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-npzxn\"" Jul 27 23:25:28.512482 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.512465 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 23:25:28.518355 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.518334 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5946b997db-jp7t2"] Jul 27 23:25:28.610593 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.610560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54845aaf-3348-425d-b57e-3b1684c5216c-tls-certs\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.610593 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.610601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54845aaf-3348-425d-b57e-3b1684c5216c-tmp\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.610915 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.610624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfkz\" (UniqueName: \"kubernetes.io/projected/54845aaf-3348-425d-b57e-3b1684c5216c-kube-api-access-4cfkz\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.711422 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.711339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54845aaf-3348-425d-b57e-3b1684c5216c-tls-certs\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.711422 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.711383 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54845aaf-3348-425d-b57e-3b1684c5216c-tmp\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.711422 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.711403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfkz\" (UniqueName: \"kubernetes.io/projected/54845aaf-3348-425d-b57e-3b1684c5216c-kube-api-access-4cfkz\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.711696 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:25:28.711488 2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 27 23:25:28.711696 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:25:28.711565 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54845aaf-3348-425d-b57e-3b1684c5216c-tls-certs podName:54845aaf-3348-425d-b57e-3b1684c5216c nodeName:}" failed. No retries permitted until 2026-07-27 23:25:29.211545372 +0000 UTC m=+572.009509973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/54845aaf-3348-425d-b57e-3b1684c5216c-tls-certs") pod "kube-auth-proxy-5946b997db-jp7t2" (UID: "54845aaf-3348-425d-b57e-3b1684c5216c") : secret "kube-auth-proxy-tls" not found Jul 27 23:25:28.713785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.713767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54845aaf-3348-425d-b57e-3b1684c5216c-tmp\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.728207 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.728182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfkz\" (UniqueName: \"kubernetes.io/projected/54845aaf-3348-425d-b57e-3b1684c5216c-kube-api-access-4cfkz\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:28.781651 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.781617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:28.878666 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.878638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:28.980033 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:28.979935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:29.087372 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.087338 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:29.217529 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.217483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54845aaf-3348-425d-b57e-3b1684c5216c-tls-certs\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:29.220214 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.220188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54845aaf-3348-425d-b57e-3b1684c5216c-tls-certs\") pod \"kube-auth-proxy-5946b997db-jp7t2\" (UID: \"54845aaf-3348-425d-b57e-3b1684c5216c\") " pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:29.415982 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.415930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:29.419998 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.419969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" Jul 27 23:25:29.564107 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.564060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:29.564736 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.564713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5946b997db-jp7t2"] Jul 27 23:25:29.566952 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:25:29.566924 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54845aaf_3348_425d_b57e_3b1684c5216c.slice/crio-d367da727424cba3067197b8ec03b7c14405513a9d76e0f659ecd2b4ca4c4fc9 WatchSource:0}: Error finding container d367da727424cba3067197b8ec03b7c14405513a9d76e0f659ecd2b4ca4c4fc9: Status 404 returned error can't find the container with id d367da727424cba3067197b8ec03b7c14405513a9d76e0f659ecd2b4ca4c4fc9 Jul 27 23:25:29.724551 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.724471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:29.790779 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.790724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:29.986318 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:29.986223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" event={"ID":"54845aaf-3348-425d-b57e-3b1684c5216c","Type":"ContainerStarted","Data":"d367da727424cba3067197b8ec03b7c14405513a9d76e0f659ecd2b4ca4c4fc9"} Jul 27 23:25:30.237021 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:30.236900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:30.628320 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:30.628280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:31.998227 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:31.998194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" event={"ID":"54845aaf-3348-425d-b57e-3b1684c5216c","Type":"ContainerStarted","Data":"2c717244426d53b57032e535573df5ed0ddf29fbab57cff21aba098276526246"} Jul 27 23:25:32.013468 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:32.013348 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5946b997db-jp7t2" podStartSLOduration=1.674179732 podStartE2EDuration="4.013328291s" totalImagesPullingTime="2.339148559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:25:32.012131477 +0000 UTC m=+574.810096103" watchObservedRunningTime="2026-07-27 23:25:32.013328291 +0000 UTC m=+574.811292916" Jul 27 23:25:36.588156 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:36.588114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:36.706778 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:36.706742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:37.269437 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:37.269407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:37.516780 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:37.516746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:37.740878 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:37.740785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:39.489046 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:39.489014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:39.615773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:39.615740 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:39.921061 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:39.921032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:40.709309 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:40.709277 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:40.810269 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:40.810229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:40.911013 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:40.910975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:25:41.654373 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:25:41.654335 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:26:29.482932 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:29.482880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:26:30.897855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:30.897790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:26:44.318972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.318920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:26:44.708976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.708899 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6"] Jul 27 23:26:44.712355 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.712340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.714515 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.714491 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 23:26:44.715227 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.715212 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 23:26:44.715328 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.715253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-27gkk\"" Jul 27 23:26:44.718783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.718746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6"] Jul 27 23:26:44.810174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.810137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbhf\" (UniqueName: \"kubernetes.io/projected/d2e9575e-9b0b-4de2-a599-412468b90a31-kube-api-access-wkbhf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.810174 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.810176 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.810390 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.810204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.911141 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.911106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbhf\" (UniqueName: \"kubernetes.io/projected/d2e9575e-9b0b-4de2-a599-412468b90a31-kube-api-access-wkbhf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.911141 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.911142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.911383 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.911169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.911519 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.911501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.911569 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.911529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:44.919098 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:44.919079 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbhf\" (UniqueName: \"kubernetes.io/projected/d2e9575e-9b0b-4de2-a599-412468b90a31-kube-api-access-wkbhf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:45.022046 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.022016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:45.151882 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.151857 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6"] Jul 27 23:26:45.154337 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:26:45.154301 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e9575e_9b0b_4de2_a599_412468b90a31.slice/crio-99bd1135021c0a27babd65b20b38cbf5039e629c6accf71a020a7620310915e4 WatchSource:0}: Error finding container 99bd1135021c0a27babd65b20b38cbf5039e629c6accf71a020a7620310915e4: Status 404 returned error can't find the container with id 99bd1135021c0a27babd65b20b38cbf5039e629c6accf71a020a7620310915e4 Jul 27 23:26:45.312106 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.312034 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579"] Jul 27 23:26:45.315442 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.315423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.322346 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.322276 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579"] Jul 27 23:26:45.352617 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.352585 2564 generic.go:356] "Generic (PLEG): container finished" podID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerID="c9a88b4dbbf8ccf5645d8c2cd5d51e5da618ac203c5c78be229c90a66208a0f0" exitCode=0 Jul 27 23:26:45.352770 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.352671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" event={"ID":"d2e9575e-9b0b-4de2-a599-412468b90a31","Type":"ContainerDied","Data":"c9a88b4dbbf8ccf5645d8c2cd5d51e5da618ac203c5c78be229c90a66208a0f0"} Jul 27 23:26:45.352770 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.352714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" event={"ID":"d2e9575e-9b0b-4de2-a599-412468b90a31","Type":"ContainerStarted","Data":"99bd1135021c0a27babd65b20b38cbf5039e629c6accf71a020a7620310915e4"} Jul 27 23:26:45.416474 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.416433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.416664 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.416487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.416664 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.416587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sf22\" (UniqueName: \"kubernetes.io/projected/a9895218-e942-4183-be21-8ac1ea0c4a67-kube-api-access-2sf22\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.517396 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.517357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.517599 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.517402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sf22\" (UniqueName: \"kubernetes.io/projected/a9895218-e942-4183-be21-8ac1ea0c4a67-kube-api-access-2sf22\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.517599 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.517490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.517809 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.517785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.517922 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.517878 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.524855 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.524833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sf22\" (UniqueName: \"kubernetes.io/projected/a9895218-e942-4183-be21-8ac1ea0c4a67-kube-api-access-2sf22\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.624975 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.624894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:45.719664 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.719629 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8"] Jul 27 23:26:45.723394 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.723373 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.732015 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.730739 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8"] Jul 27 23:26:45.788089 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.788050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579"] Jul 27 23:26:45.798933 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:26:45.798901 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9895218_e942_4183_be21_8ac1ea0c4a67.slice/crio-f30fd015b65d5baaa8283d06f487b708520e0f84949afad677daf91dcea9e7a8 WatchSource:0}: Error finding container f30fd015b65d5baaa8283d06f487b708520e0f84949afad677daf91dcea9e7a8: Status 404 returned error can't find the container with id f30fd015b65d5baaa8283d06f487b708520e0f84949afad677daf91dcea9e7a8 Jul 27 23:26:45.819917 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.819886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.820029 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.819934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.820029 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.819964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxjx\" (UniqueName: \"kubernetes.io/projected/87ed1365-7594-4eee-a8b4-f4e597ceb656-kube-api-access-7mxjx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.920753 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.920721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.920891 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.920758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.920891 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.920779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxjx\" (UniqueName: \"kubernetes.io/projected/87ed1365-7594-4eee-a8b4-f4e597ceb656-kube-api-access-7mxjx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.921173 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.921149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.921209 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.921167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:45.928436 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:45.928410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxjx\" (UniqueName: \"kubernetes.io/projected/87ed1365-7594-4eee-a8b4-f4e597ceb656-kube-api-access-7mxjx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:46.038020 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.037986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:46.114766 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.114734 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7"] Jul 27 23:26:46.118546 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.118524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.125752 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.125720 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7"] Jul 27 23:26:46.223912 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.223886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.224024 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.223935 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.224087 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.224060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9cj\" (UniqueName: \"kubernetes.io/projected/5cff4401-e346-4a76-a007-cb4f72e63dce-kube-api-access-2f9cj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.277933 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.277898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8"] Jul 27 23:26:46.281152 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:26:46.281126 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ed1365_7594_4eee_a8b4_f4e597ceb656.slice/crio-459a526f4fdfe114c92cc4a2d2bf29725d6091bfb504985d0da1e8fb94e8f480 WatchSource:0}: Error finding container 459a526f4fdfe114c92cc4a2d2bf29725d6091bfb504985d0da1e8fb94e8f480: Status 404 returned error can't find the container with id 459a526f4fdfe114c92cc4a2d2bf29725d6091bfb504985d0da1e8fb94e8f480 Jul 27 23:26:46.324757 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.324718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.325145 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.324766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.325145 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.324861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9cj\" (UniqueName: \"kubernetes.io/projected/5cff4401-e346-4a76-a007-cb4f72e63dce-kube-api-access-2f9cj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.325145 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.325131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.325263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.325160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.333438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.333408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9cj\" (UniqueName: \"kubernetes.io/projected/5cff4401-e346-4a76-a007-cb4f72e63dce-kube-api-access-2f9cj\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.359342 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.359312 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerID="d8dc699a45953b9918f19e794455b37c15dc6a8ff40f61aea106528509aba089" exitCode=0 Jul 27 23:26:46.359495 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.359386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" event={"ID":"a9895218-e942-4183-be21-8ac1ea0c4a67","Type":"ContainerDied","Data":"d8dc699a45953b9918f19e794455b37c15dc6a8ff40f61aea106528509aba089"} Jul 27 23:26:46.359495 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.359423 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" event={"ID":"a9895218-e942-4183-be21-8ac1ea0c4a67","Type":"ContainerStarted","Data":"f30fd015b65d5baaa8283d06f487b708520e0f84949afad677daf91dcea9e7a8"} Jul 27 23:26:46.360684 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.360660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" event={"ID":"87ed1365-7594-4eee-a8b4-f4e597ceb656","Type":"ContainerStarted","Data":"459a526f4fdfe114c92cc4a2d2bf29725d6091bfb504985d0da1e8fb94e8f480"} Jul 27 23:26:46.362502 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.362475 2564 generic.go:356] "Generic (PLEG): container finished" podID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerID="ab168115ca617795b566c588725a7c1ba7a936ec32f4f7d8e3303a6b39120f09" exitCode=0 Jul 27 23:26:46.362627 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.362537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" event={"ID":"d2e9575e-9b0b-4de2-a599-412468b90a31","Type":"ContainerDied","Data":"ab168115ca617795b566c588725a7c1ba7a936ec32f4f7d8e3303a6b39120f09"} Jul 27 23:26:46.429613 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.429583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:46.566943 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:46.566918 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7"] Jul 27 23:26:46.568918 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:26:46.568891 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cff4401_e346_4a76_a007_cb4f72e63dce.slice/crio-095526ed59e6d500b77515e0564fa3cc5b7d151ff8fb027800854651cf248962 WatchSource:0}: Error finding container 095526ed59e6d500b77515e0564fa3cc5b7d151ff8fb027800854651cf248962: Status 404 returned error can't find the container with id 095526ed59e6d500b77515e0564fa3cc5b7d151ff8fb027800854651cf248962 Jul 27 23:26:47.369698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.369662 2564 generic.go:356] "Generic (PLEG): container finished" podID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerID="fece331c1fb90225be373032cd40f23e1336726a701a314167dfc8392bdc54be" exitCode=0 Jul 27 23:26:47.370159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.369769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" event={"ID":"5cff4401-e346-4a76-a007-cb4f72e63dce","Type":"ContainerDied","Data":"fece331c1fb90225be373032cd40f23e1336726a701a314167dfc8392bdc54be"} Jul 27 23:26:47.370159 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.369856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" event={"ID":"5cff4401-e346-4a76-a007-cb4f72e63dce","Type":"ContainerStarted","Data":"095526ed59e6d500b77515e0564fa3cc5b7d151ff8fb027800854651cf248962"} Jul 27 23:26:47.371674 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.371652 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerID="7dd039748353c4e46ec38d885e8490975a013ff14a3d3e07b1e862cd645616e4" exitCode=0 Jul 27 23:26:47.371778 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.371738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" event={"ID":"a9895218-e942-4183-be21-8ac1ea0c4a67","Type":"ContainerDied","Data":"7dd039748353c4e46ec38d885e8490975a013ff14a3d3e07b1e862cd645616e4"} Jul 27 23:26:47.373459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.373439 2564 generic.go:356] "Generic (PLEG): container finished" podID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerID="0d1e9bab0c029f14691444fd9f3adaf5e9680c0f59f73039556e6eb6a28c9aa4" exitCode=0 Jul 27 23:26:47.373563 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.373514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" event={"ID":"87ed1365-7594-4eee-a8b4-f4e597ceb656","Type":"ContainerDied","Data":"0d1e9bab0c029f14691444fd9f3adaf5e9680c0f59f73039556e6eb6a28c9aa4"} Jul 27 23:26:47.375864 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.375842 2564 generic.go:356] "Generic (PLEG): container finished" podID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerID="4963d0b56bd39657658550fa8495f4ac85728a53702dff22a4f422fc966d9328" exitCode=0 Jul 27 23:26:47.375954 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:47.375873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" event={"ID":"d2e9575e-9b0b-4de2-a599-412468b90a31","Type":"ContainerDied","Data":"4963d0b56bd39657658550fa8495f4ac85728a53702dff22a4f422fc966d9328"} Jul 27 23:26:48.383300 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.383261 2564 generic.go:356] "Generic (PLEG): container finished" podID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerID="4a67d973263323d678d39c3560469d63d952cb90e38b4b1ac79a61309ce9ae6e" exitCode=0 Jul 27 23:26:48.383683 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.383319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" event={"ID":"5cff4401-e346-4a76-a007-cb4f72e63dce","Type":"ContainerDied","Data":"4a67d973263323d678d39c3560469d63d952cb90e38b4b1ac79a61309ce9ae6e"} Jul 27 23:26:48.385485 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.385462 2564 generic.go:356] "Generic (PLEG): container finished" podID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerID="0244183299af06f42ef539ff22175d232bec5266c0a341a7d875d18ef06bb0b2" exitCode=0 Jul 27 23:26:48.385587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.385503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" event={"ID":"a9895218-e942-4183-be21-8ac1ea0c4a67","Type":"ContainerDied","Data":"0244183299af06f42ef539ff22175d232bec5266c0a341a7d875d18ef06bb0b2"} Jul 27 23:26:48.387327 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.387308 2564 generic.go:356] "Generic (PLEG): container finished" podID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerID="4ccd431299b684a998bddffe40273db8a71400241d81c4b882ab779af9646d11" exitCode=0 Jul 27 23:26:48.387396 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.387365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" event={"ID":"87ed1365-7594-4eee-a8b4-f4e597ceb656","Type":"ContainerDied","Data":"4ccd431299b684a998bddffe40273db8a71400241d81c4b882ab779af9646d11"} Jul 27 23:26:48.520994 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.520967 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:48.542931 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.542902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkbhf\" (UniqueName: \"kubernetes.io/projected/d2e9575e-9b0b-4de2-a599-412468b90a31-kube-api-access-wkbhf\") pod \"d2e9575e-9b0b-4de2-a599-412468b90a31\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " Jul 27 23:26:48.543076 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.542942 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-util\") pod \"d2e9575e-9b0b-4de2-a599-412468b90a31\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " Jul 27 23:26:48.543076 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.542980 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-bundle\") pod \"d2e9575e-9b0b-4de2-a599-412468b90a31\" (UID: \"d2e9575e-9b0b-4de2-a599-412468b90a31\") " Jul 27 23:26:48.543535 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.543495 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-bundle" (OuterVolumeSpecName: "bundle") pod "d2e9575e-9b0b-4de2-a599-412468b90a31" (UID: "d2e9575e-9b0b-4de2-a599-412468b90a31"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:48.545727 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.545699 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e9575e-9b0b-4de2-a599-412468b90a31-kube-api-access-wkbhf" (OuterVolumeSpecName: "kube-api-access-wkbhf") pod "d2e9575e-9b0b-4de2-a599-412468b90a31" (UID: "d2e9575e-9b0b-4de2-a599-412468b90a31"). InnerVolumeSpecName "kube-api-access-wkbhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:26:48.549403 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.549378 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-util" (OuterVolumeSpecName: "util") pod "d2e9575e-9b0b-4de2-a599-412468b90a31" (UID: "d2e9575e-9b0b-4de2-a599-412468b90a31"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:48.643874 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.643847 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:48.643874 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.643873 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkbhf\" (UniqueName: \"kubernetes.io/projected/d2e9575e-9b0b-4de2-a599-412468b90a31-kube-api-access-wkbhf\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:48.644032 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:48.643881 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e9575e-9b0b-4de2-a599-412468b90a31-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:49.396556 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.396523 2564 generic.go:356] "Generic (PLEG): container finished" podID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerID="1113149979ffa142c4c1349d4741c4e2f8ca976770dd61d7f5df0c1d7b11d5e1" exitCode=0 Jul 27 23:26:49.397007 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.396619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" event={"ID":"5cff4401-e346-4a76-a007-cb4f72e63dce","Type":"ContainerDied","Data":"1113149979ffa142c4c1349d4741c4e2f8ca976770dd61d7f5df0c1d7b11d5e1"} Jul 27 23:26:49.398661 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.398639 2564 generic.go:356] "Generic (PLEG): container finished" podID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerID="210a4d56f26927078895c698b5d1b88bf3a163b5473cb5341087adba456f060c" exitCode=0 Jul 27 23:26:49.398769 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.398717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" event={"ID":"87ed1365-7594-4eee-a8b4-f4e597ceb656","Type":"ContainerDied","Data":"210a4d56f26927078895c698b5d1b88bf3a163b5473cb5341087adba456f060c"} Jul 27 23:26:49.400497 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.400476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" event={"ID":"d2e9575e-9b0b-4de2-a599-412468b90a31","Type":"ContainerDied","Data":"99bd1135021c0a27babd65b20b38cbf5039e629c6accf71a020a7620310915e4"} Jul 27 23:26:49.406832 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.400772 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99bd1135021c0a27babd65b20b38cbf5039e629c6accf71a020a7620310915e4" Jul 27 23:26:49.406832 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.401339 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6" Jul 27 23:26:49.536217 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.536193 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:49.654137 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.654098 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-util\") pod \"a9895218-e942-4183-be21-8ac1ea0c4a67\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " Jul 27 23:26:49.654319 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.654198 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-bundle\") pod \"a9895218-e942-4183-be21-8ac1ea0c4a67\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " Jul 27 23:26:49.654319 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.654226 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sf22\" (UniqueName: \"kubernetes.io/projected/a9895218-e942-4183-be21-8ac1ea0c4a67-kube-api-access-2sf22\") pod \"a9895218-e942-4183-be21-8ac1ea0c4a67\" (UID: \"a9895218-e942-4183-be21-8ac1ea0c4a67\") " Jul 27 23:26:49.654748 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.654722 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-bundle" (OuterVolumeSpecName: "bundle") pod "a9895218-e942-4183-be21-8ac1ea0c4a67" (UID: "a9895218-e942-4183-be21-8ac1ea0c4a67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:49.656726 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.656698 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9895218-e942-4183-be21-8ac1ea0c4a67-kube-api-access-2sf22" (OuterVolumeSpecName: "kube-api-access-2sf22") pod "a9895218-e942-4183-be21-8ac1ea0c4a67" (UID: "a9895218-e942-4183-be21-8ac1ea0c4a67"). InnerVolumeSpecName "kube-api-access-2sf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:26:49.659462 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.659423 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-util" (OuterVolumeSpecName: "util") pod "a9895218-e942-4183-be21-8ac1ea0c4a67" (UID: "a9895218-e942-4183-be21-8ac1ea0c4a67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:49.755265 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.755242 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:49.755381 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.755267 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sf22\" (UniqueName: \"kubernetes.io/projected/a9895218-e942-4183-be21-8ac1ea0c4a67-kube-api-access-2sf22\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:49.755381 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:49.755277 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9895218-e942-4183-be21-8ac1ea0c4a67-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:50.406885 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.406857 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" Jul 27 23:26:50.406885 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.406870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579" event={"ID":"a9895218-e942-4183-be21-8ac1ea0c4a67","Type":"ContainerDied","Data":"f30fd015b65d5baaa8283d06f487b708520e0f84949afad677daf91dcea9e7a8"} Jul 27 23:26:50.407326 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.406912 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f30fd015b65d5baaa8283d06f487b708520e0f84949afad677daf91dcea9e7a8" Jul 27 23:26:50.540204 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.540180 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:50.560894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.560869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-util\") pod \"5cff4401-e346-4a76-a007-cb4f72e63dce\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " Jul 27 23:26:50.561019 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.560918 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9cj\" (UniqueName: \"kubernetes.io/projected/5cff4401-e346-4a76-a007-cb4f72e63dce-kube-api-access-2f9cj\") pod \"5cff4401-e346-4a76-a007-cb4f72e63dce\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " Jul 27 23:26:50.561019 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.560960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-bundle\") pod \"5cff4401-e346-4a76-a007-cb4f72e63dce\" (UID: \"5cff4401-e346-4a76-a007-cb4f72e63dce\") " Jul 27 23:26:50.561578 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.561539 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-bundle" (OuterVolumeSpecName: "bundle") pod "5cff4401-e346-4a76-a007-cb4f72e63dce" (UID: "5cff4401-e346-4a76-a007-cb4f72e63dce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:50.563189 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.563154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cff4401-e346-4a76-a007-cb4f72e63dce-kube-api-access-2f9cj" (OuterVolumeSpecName: "kube-api-access-2f9cj") pod "5cff4401-e346-4a76-a007-cb4f72e63dce" (UID: "5cff4401-e346-4a76-a007-cb4f72e63dce"). InnerVolumeSpecName "kube-api-access-2f9cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:26:50.569916 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.569890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-util" (OuterVolumeSpecName: "util") pod "5cff4401-e346-4a76-a007-cb4f72e63dce" (UID: "5cff4401-e346-4a76-a007-cb4f72e63dce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:50.579554 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.579534 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:50.661542 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.661478 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mxjx\" (UniqueName: \"kubernetes.io/projected/87ed1365-7594-4eee-a8b4-f4e597ceb656-kube-api-access-7mxjx\") pod \"87ed1365-7594-4eee-a8b4-f4e597ceb656\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " Jul 27 23:26:50.661542 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.661527 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-util\") pod \"87ed1365-7594-4eee-a8b4-f4e597ceb656\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " Jul 27 23:26:50.661739 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.661571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-bundle\") pod \"87ed1365-7594-4eee-a8b4-f4e597ceb656\" (UID: \"87ed1365-7594-4eee-a8b4-f4e597ceb656\") " Jul 27 23:26:50.661833 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.661800 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:50.661906 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.661847 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9cj\" (UniqueName: \"kubernetes.io/projected/5cff4401-e346-4a76-a007-cb4f72e63dce-kube-api-access-2f9cj\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:50.661906 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.661862 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cff4401-e346-4a76-a007-cb4f72e63dce-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:50.662125 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.662094 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-bundle" (OuterVolumeSpecName: "bundle") pod "87ed1365-7594-4eee-a8b4-f4e597ceb656" (UID: "87ed1365-7594-4eee-a8b4-f4e597ceb656"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:50.663686 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.663667 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ed1365-7594-4eee-a8b4-f4e597ceb656-kube-api-access-7mxjx" (OuterVolumeSpecName: "kube-api-access-7mxjx") pod "87ed1365-7594-4eee-a8b4-f4e597ceb656" (UID: "87ed1365-7594-4eee-a8b4-f4e597ceb656"). InnerVolumeSpecName "kube-api-access-7mxjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:26:50.667207 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.667184 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-util" (OuterVolumeSpecName: "util") pod "87ed1365-7594-4eee-a8b4-f4e597ceb656" (UID: "87ed1365-7594-4eee-a8b4-f4e597ceb656"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:26:50.762691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.762665 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxjx\" (UniqueName: \"kubernetes.io/projected/87ed1365-7594-4eee-a8b4-f4e597ceb656-kube-api-access-7mxjx\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:50.762691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.762687 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-util\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:50.762841 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:50.762696 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87ed1365-7594-4eee-a8b4-f4e597ceb656-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:26:51.413894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:51.413863 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" Jul 27 23:26:51.413894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:51.413870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7" event={"ID":"5cff4401-e346-4a76-a007-cb4f72e63dce","Type":"ContainerDied","Data":"095526ed59e6d500b77515e0564fa3cc5b7d151ff8fb027800854651cf248962"} Jul 27 23:26:51.414399 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:51.413921 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="095526ed59e6d500b77515e0564fa3cc5b7d151ff8fb027800854651cf248962" Jul 27 23:26:51.415723 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:51.415704 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" Jul 27 23:26:51.415896 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:51.415704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8" event={"ID":"87ed1365-7594-4eee-a8b4-f4e597ceb656","Type":"ContainerDied","Data":"459a526f4fdfe114c92cc4a2d2bf29725d6091bfb504985d0da1e8fb94e8f480"} Jul 27 23:26:51.415896 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:51.415810 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="459a526f4fdfe114c92cc4a2d2bf29725d6091bfb504985d0da1e8fb94e8f480" Jul 27 23:26:58.089976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:26:58.089943 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:01.696322 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:01.696291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:01.802567 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:01.802530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:03.480190 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480153 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58598dcbf4-9kgvk"] Jul 27 23:27:03.480658 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480610 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="util" Jul 27 23:27:03.480658 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480626 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="util" Jul 27 23:27:03.480658 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480639 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="util" Jul 27 23:27:03.480658 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480646 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="util" Jul 27 23:27:03.480658 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480658 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480666 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480677 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="util" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480685 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="util" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480704 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480712 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480731 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480739 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480747 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480755 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480769 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480777 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480788 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480795 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480804 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480833 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480843 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480852 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="pull" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480862 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="util" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480870 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="util" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480953 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9895218-e942-4183-be21-8ac1ea0c4a67" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480965 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="87ed1365-7594-4eee-a8b4-f4e597ceb656" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480976 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cff4401-e346-4a76-a007-cb4f72e63dce" containerName="extract" Jul 27 23:27:03.480977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.480984 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e9575e-9b0b-4de2-a599-412468b90a31" containerName="extract" Jul 27 23:27:03.484588 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.484565 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.492994 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.492971 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58598dcbf4-9kgvk"] Jul 27 23:27:03.533173 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.533145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:03.568412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-trusted-ca-bundle\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.568575 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-oauth-serving-cert\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.568575 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-service-ca\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.568575 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5929e6-0cc1-4dd5-8467-f83847af5370-console-serving-cert\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.568575 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5929e6-0cc1-4dd5-8467-f83847af5370-console-oauth-config\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.568575 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-console-config\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.568756 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.568588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9crb\" (UniqueName: \"kubernetes.io/projected/da5929e6-0cc1-4dd5-8467-f83847af5370-kube-api-access-j9crb\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.640138 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.640101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:03.669415 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-trusted-ca-bundle\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.669580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-oauth-serving-cert\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.669580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-service-ca\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.669580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5929e6-0cc1-4dd5-8467-f83847af5370-console-serving-cert\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.669580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5929e6-0cc1-4dd5-8467-f83847af5370-console-oauth-config\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.669580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-console-config\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.669580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.669548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9crb\" (UniqueName: \"kubernetes.io/projected/da5929e6-0cc1-4dd5-8467-f83847af5370-kube-api-access-j9crb\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.670237 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.670214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-oauth-serving-cert\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.670399 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.670253 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-console-config\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.670399 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.670269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-service-ca\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.670582 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.670562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5929e6-0cc1-4dd5-8467-f83847af5370-trusted-ca-bundle\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.672099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.672083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5929e6-0cc1-4dd5-8467-f83847af5370-console-serving-cert\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.672237 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.672219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5929e6-0cc1-4dd5-8467-f83847af5370-console-oauth-config\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.681467 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.681434 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9crb\" (UniqueName: \"kubernetes.io/projected/da5929e6-0cc1-4dd5-8467-f83847af5370-kube-api-access-j9crb\") pod \"console-58598dcbf4-9kgvk\" (UID: \"da5929e6-0cc1-4dd5-8467-f83847af5370\") " pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.794981 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.794951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:03.928260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:03.928234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58598dcbf4-9kgvk"] Jul 27 23:27:03.930228 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:27:03.930203 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5929e6_0cc1_4dd5_8467_f83847af5370.slice/crio-323f45c8d14b5e2169b234154c5b30ce97d3d8d0e396bdc55893d90f74c954be WatchSource:0}: Error finding container 323f45c8d14b5e2169b234154c5b30ce97d3d8d0e396bdc55893d90f74c954be: Status 404 returned error can't find the container with id 323f45c8d14b5e2169b234154c5b30ce97d3d8d0e396bdc55893d90f74c954be Jul 27 23:27:04.317460 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:04.317421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:04.481111 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:04.481072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58598dcbf4-9kgvk" event={"ID":"da5929e6-0cc1-4dd5-8467-f83847af5370","Type":"ContainerStarted","Data":"6ce6e5607bfcbf313edc3ec2d15fa86a0227811a6d497a8cf7d626df34936bd5"} Jul 27 23:27:04.481539 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:04.481134 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58598dcbf4-9kgvk" event={"ID":"da5929e6-0cc1-4dd5-8467-f83847af5370","Type":"ContainerStarted","Data":"323f45c8d14b5e2169b234154c5b30ce97d3d8d0e396bdc55893d90f74c954be"} Jul 27 23:27:04.497360 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:04.497309 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58598dcbf4-9kgvk" podStartSLOduration=1.497293354 podStartE2EDuration="1.497293354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:04.496615866 +0000 UTC m=+667.294580490" watchObservedRunningTime="2026-07-27 23:27:04.497293354 +0000 UTC m=+667.295257978" Jul 27 23:27:06.709317 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:06.709272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:07.127438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:07.127408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:09.303736 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:09.303696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:09.609392 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:09.609310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:09.708591 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:09.708557 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:11.925324 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:11.925290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:12.030431 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:12.030400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:13.795637 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:13.795610 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:13.795637 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:13.795646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:13.800119 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:13.800097 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:13.897015 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:13.896991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:13.996882 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:13.996858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:14.542094 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:14.542066 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58598dcbf4-9kgvk" Jul 27 23:27:14.589038 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:14.589003 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5c99cd54-tbnrc"] Jul 27 23:27:14.689353 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:14.689323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:14.789317 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:14.789291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:14.889593 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:14.889523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:15.483487 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.483456 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq"] Jul 27 23:27:15.486943 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.486926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.489726 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.489705 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 23:27:15.489882 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.489796 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 23:27:15.489882 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.489860 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-27gkk\"" Jul 27 23:27:15.490678 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.490663 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 23:27:15.490762 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.490677 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 23:27:15.504651 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.504627 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq"] Jul 27 23:27:15.519054 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.519026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:15.574637 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.574611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.574742 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.574644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84tt\" (UniqueName: \"kubernetes.io/projected/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-kube-api-access-h84tt\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.574742 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.574731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.642406 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.642378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:15.675618 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.675590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.675730 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.675673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.675802 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.675739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84tt\" (UniqueName: \"kubernetes.io/projected/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-kube-api-access-h84tt\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.676480 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.676460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.678223 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.678194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.683629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.683611 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84tt\" (UniqueName: \"kubernetes.io/projected/c1ab9c4f-0c36-4cbe-837f-1b3208c69a75-kube-api-access-h84tt\") pod \"kuadrant-console-plugin-66b5774985-qxbhq\" (UID: \"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.796748 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.796723 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" Jul 27 23:27:15.924226 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.924200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq"] Jul 27 23:27:15.926165 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:27:15.926125 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ab9c4f_0c36_4cbe_837f_1b3208c69a75.slice/crio-07736750d3cf692a0ab1ef4ab3483649daaed46a04ab9afdc20a7883a77d9c7e WatchSource:0}: Error finding container 07736750d3cf692a0ab1ef4ab3483649daaed46a04ab9afdc20a7883a77d9c7e: Status 404 returned error can't find the container with id 07736750d3cf692a0ab1ef4ab3483649daaed46a04ab9afdc20a7883a77d9c7e Jul 27 23:27:15.927456 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:15.927440 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 23:27:16.549230 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:16.549199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" event={"ID":"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75","Type":"ContainerStarted","Data":"07736750d3cf692a0ab1ef4ab3483649daaed46a04ab9afdc20a7883a77d9c7e"} Jul 27 23:27:17.093951 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:17.093882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:17.193932 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:17.193899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:17.986293 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:17.986264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:18.087070 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:18.087020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:19.393350 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:19.393308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:19.495409 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:19.495380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:23.345785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:23.345746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:23.395977 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:23.395942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:29.910636 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:29.910579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:30.018830 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:30.018773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:32.501025 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:32.500989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:32.634215 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:32.634180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:35.504023 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:35.503985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:35.558768 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:35.558739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:38.684104 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:38.684063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" event={"ID":"c1ab9c4f-0c36-4cbe-837f-1b3208c69a75","Type":"ContainerStarted","Data":"66e8f65ee8569fdd8f5130f08167f275461adb38a583dcb02e476fa1d7d885db"} Jul 27 23:27:38.698909 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:38.698855 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-qxbhq" podStartSLOduration=1.39175435 podStartE2EDuration="23.698791594s" totalImagesPullingTime="22.307037244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:27:38.698475744 +0000 UTC m=+701.496440369" watchObservedRunningTime="2026-07-27 23:27:38.698791594 +0000 UTC m=+701.496756219" Jul 27 23:27:39.610457 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.610418 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c5c99cd54-tbnrc" podUID="47b38bdc-82fa-4b25-8f18-3955deaf9f38" containerName="console" containerID="cri-o://562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b" gracePeriod=15 Jul 27 23:27:39.869499 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.869446 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:27:39.898841 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.898799 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-service-ca\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.898975 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.898868 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-config\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.898975 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.898898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-trusted-ca-bundle\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.898975 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.898953 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-serving-cert\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.899141 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.898988 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-oauth-config\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.899141 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.899015 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-oauth-serving-cert\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.899141 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.899054 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vh26\" (UniqueName: \"kubernetes.io/projected/47b38bdc-82fa-4b25-8f18-3955deaf9f38-kube-api-access-7vh26\") pod \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\" (UID: \"47b38bdc-82fa-4b25-8f18-3955deaf9f38\") " Jul 27 23:27:39.899414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.899323 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-config" (OuterVolumeSpecName: "console-config") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:27:39.899477 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.899405 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:27:39.899566 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.899543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:27:39.899566 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.899558 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-service-ca" (OuterVolumeSpecName: "service-ca") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 23:27:39.901479 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.901454 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:27:39.901566 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.901505 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b38bdc-82fa-4b25-8f18-3955deaf9f38-kube-api-access-7vh26" (OuterVolumeSpecName: "kube-api-access-7vh26") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "kube-api-access-7vh26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:27:39.901774 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:39.901756 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "47b38bdc-82fa-4b25-8f18-3955deaf9f38" (UID: "47b38bdc-82fa-4b25-8f18-3955deaf9f38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:27:40.000149 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000114 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.000149 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000142 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.000149 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000151 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-trusted-ca-bundle\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.000149 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000160 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.000410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000169 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/47b38bdc-82fa-4b25-8f18-3955deaf9f38-console-oauth-config\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.000410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000177 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/47b38bdc-82fa-4b25-8f18-3955deaf9f38-oauth-serving-cert\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.000410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.000186 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vh26\" (UniqueName: \"kubernetes.io/projected/47b38bdc-82fa-4b25-8f18-3955deaf9f38-kube-api-access-7vh26\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:27:40.701089 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.701050 2564 generic.go:356] "Generic (PLEG): container finished" podID="47b38bdc-82fa-4b25-8f18-3955deaf9f38" containerID="562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b" exitCode=0 Jul 27 23:27:40.701260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.701104 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5c99cd54-tbnrc" Jul 27 23:27:40.701260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.701133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5c99cd54-tbnrc" event={"ID":"47b38bdc-82fa-4b25-8f18-3955deaf9f38","Type":"ContainerDied","Data":"562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b"} Jul 27 23:27:40.701260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.701216 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5c99cd54-tbnrc" event={"ID":"47b38bdc-82fa-4b25-8f18-3955deaf9f38","Type":"ContainerDied","Data":"4f38796d558c6babaa154cce36e386485119302826a657fe54014c2aece5a158"} Jul 27 23:27:40.701260 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.701242 2564 scope.go:117] "RemoveContainer" containerID="562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b" Jul 27 23:27:40.710410 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.710394 2564 scope.go:117] "RemoveContainer" containerID="562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b" Jul 27 23:27:40.710662 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:27:40.710641 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b\": container with ID starting with 562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b not found: ID does not exist" containerID="562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b" Jul 27 23:27:40.710725 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.710673 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b"} err="failed to get container status \"562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b\": rpc error: code = NotFound desc = could not find container \"562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b\": container with ID starting with 562e411af7df0d9eb18c4d3ea9b528acd104e16aee4b39dce881c22d2968659b not found: ID does not exist" Jul 27 23:27:40.721264 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.721241 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5c99cd54-tbnrc"] Jul 27 23:27:40.724282 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:40.724262 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c5c99cd54-tbnrc"] Jul 27 23:27:41.782641 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:41.782612 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b38bdc-82fa-4b25-8f18-3955deaf9f38" path="/var/lib/kubelet/pods/47b38bdc-82fa-4b25-8f18-3955deaf9f38/volumes" Jul 27 23:27:42.900695 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:42.900651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:42.949652 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:42.949617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:48.711265 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:48.711186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:48.810879 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:48.810849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:53.608728 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:53.608693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:53.658654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:53.658630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:55.542589 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:55.542552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:55.691058 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:55.691027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:57.287425 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.287388 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:57.392914 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.392867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:57.444262 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.444235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:57.754239 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.754207 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:27:57.754615 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.754589 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b38bdc-82fa-4b25-8f18-3955deaf9f38" containerName="console" Jul 27 23:27:57.754615 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.754609 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b38bdc-82fa-4b25-8f18-3955deaf9f38" containerName="console" Jul 27 23:27:57.754771 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.754696 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b38bdc-82fa-4b25-8f18-3955deaf9f38" containerName="console" Jul 27 23:27:57.772150 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.772122 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:27:57.772267 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.772239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:57.774558 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.774536 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 23:27:57.785580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.785557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:27:57.847432 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.847411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e8925ee-e527-48ae-bf8a-5ec8356053a9-config-file\") pod \"limitador-limitador-54d6d988d5-twx6m\" (UID: \"6e8925ee-e527-48ae-bf8a-5ec8356053a9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:57.847557 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.847454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26v9\" (UniqueName: \"kubernetes.io/projected/6e8925ee-e527-48ae-bf8a-5ec8356053a9-kube-api-access-l26v9\") pod \"limitador-limitador-54d6d988d5-twx6m\" (UID: \"6e8925ee-e527-48ae-bf8a-5ec8356053a9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:57.948284 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.948221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e8925ee-e527-48ae-bf8a-5ec8356053a9-config-file\") pod \"limitador-limitador-54d6d988d5-twx6m\" (UID: \"6e8925ee-e527-48ae-bf8a-5ec8356053a9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:57.948284 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.948254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26v9\" (UniqueName: \"kubernetes.io/projected/6e8925ee-e527-48ae-bf8a-5ec8356053a9-kube-api-access-l26v9\") pod \"limitador-limitador-54d6d988d5-twx6m\" (UID: \"6e8925ee-e527-48ae-bf8a-5ec8356053a9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:57.948795 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.948779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e8925ee-e527-48ae-bf8a-5ec8356053a9-config-file\") pod \"limitador-limitador-54d6d988d5-twx6m\" (UID: \"6e8925ee-e527-48ae-bf8a-5ec8356053a9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:57.955718 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:57.955691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26v9\" (UniqueName: \"kubernetes.io/projected/6e8925ee-e527-48ae-bf8a-5ec8356053a9-kube-api-access-l26v9\") pod \"limitador-limitador-54d6d988d5-twx6m\" (UID: \"6e8925ee-e527-48ae-bf8a-5ec8356053a9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:58.084226 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.084197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:27:58.113455 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.113409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:58.175345 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.175315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:27:58.198067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.198038 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-psj9h"] Jul 27 23:27:58.201713 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.201656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:27:58.222313 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.222266 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nx6j8\"" Jul 27 23:27:58.225754 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.225731 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-psj9h"] Jul 27 23:27:58.227958 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:27:58.227932 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8925ee_e527_48ae_bf8a_5ec8356053a9.slice/crio-a88beb81017b1fd0d5441a92b2e61470ebae0385c028e788c6e0da9f17447f9f WatchSource:0}: Error finding container a88beb81017b1fd0d5441a92b2e61470ebae0385c028e788c6e0da9f17447f9f: Status 404 returned error can't find the container with id a88beb81017b1fd0d5441a92b2e61470ebae0385c028e788c6e0da9f17447f9f Jul 27 23:27:58.228113 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.228092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:27:58.352991 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.352959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srppl\" (UniqueName: \"kubernetes.io/projected/c5a46d67-e91b-469d-98ab-4df863a85c1e-kube-api-access-srppl\") pod \"authorino-75d556bd77-psj9h\" (UID: \"c5a46d67-e91b-469d-98ab-4df863a85c1e\") " pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:27:58.454122 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.454047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srppl\" (UniqueName: \"kubernetes.io/projected/c5a46d67-e91b-469d-98ab-4df863a85c1e-kube-api-access-srppl\") pod \"authorino-75d556bd77-psj9h\" (UID: \"c5a46d67-e91b-469d-98ab-4df863a85c1e\") " pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:27:58.477737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.477715 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srppl\" (UniqueName: \"kubernetes.io/projected/c5a46d67-e91b-469d-98ab-4df863a85c1e-kube-api-access-srppl\") pod \"authorino-75d556bd77-psj9h\" (UID: \"c5a46d67-e91b-469d-98ab-4df863a85c1e\") " pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:27:58.537644 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.537622 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:27:58.666130 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.666102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-psj9h"] Jul 27 23:27:58.795276 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.795235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" event={"ID":"6e8925ee-e527-48ae-bf8a-5ec8356053a9","Type":"ContainerStarted","Data":"a88beb81017b1fd0d5441a92b2e61470ebae0385c028e788c6e0da9f17447f9f"} Jul 27 23:27:58.797102 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:27:58.797074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-psj9h" event={"ID":"c5a46d67-e91b-469d-98ab-4df863a85c1e","Type":"ContainerStarted","Data":"2db41e9b4aac423c845ad2e1c90ec7c49b837c5a7281f96f958f3ddc93078e2b"} Jul 27 23:28:02.825037 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:02.824998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-psj9h" event={"ID":"c5a46d67-e91b-469d-98ab-4df863a85c1e","Type":"ContainerStarted","Data":"9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289"} Jul 27 23:28:02.826798 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:02.826771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" event={"ID":"6e8925ee-e527-48ae-bf8a-5ec8356053a9","Type":"ContainerStarted","Data":"027b75b3819006c7e8d866a39749277feb61b7ae1f25e2b06b9ab5125eda5e90"} Jul 27 23:28:02.826943 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:02.826872 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:28:02.840041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:02.840003 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-psj9h" podStartSLOduration=0.828002047 podStartE2EDuration="4.839988384s" totalImagesPullingTime="4.011986337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:02.837982162 +0000 UTC m=+725.635946787" watchObservedRunningTime="2026-07-27 23:28:02.839988384 +0000 UTC m=+725.637953058" Jul 27 23:28:02.854194 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:02.854148 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" podStartSLOduration=1.401818761 podStartE2EDuration="5.85413084s" totalImagesPullingTime="4.452312079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:28:02.85199757 +0000 UTC m=+725.649962197" watchObservedRunningTime="2026-07-27 23:28:02.85413084 +0000 UTC m=+725.652095465" Jul 27 23:28:11.105299 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:11.105261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:11.159453 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:11.159421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:13.832501 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:13.832471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-twx6m" Jul 27 23:28:18.440692 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:18.440656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:18.540699 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:18.540673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:18.911958 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:18.911927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:29.120770 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:29.120738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:29.171946 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:29.171919 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:30.835589 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:30.835553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:31.338691 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:31.338651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:33.595583 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.595553 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-psj9h"] Jul 27 23:28:33.596016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.595748 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-psj9h" podUID="c5a46d67-e91b-469d-98ab-4df863a85c1e" containerName="authorino" containerID="cri-o://9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289" gracePeriod=30 Jul 27 23:28:33.852327 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.852277 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:28:33.946084 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.946058 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srppl\" (UniqueName: \"kubernetes.io/projected/c5a46d67-e91b-469d-98ab-4df863a85c1e-kube-api-access-srppl\") pod \"c5a46d67-e91b-469d-98ab-4df863a85c1e\" (UID: \"c5a46d67-e91b-469d-98ab-4df863a85c1e\") " Jul 27 23:28:33.948118 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.948092 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a46d67-e91b-469d-98ab-4df863a85c1e-kube-api-access-srppl" (OuterVolumeSpecName: "kube-api-access-srppl") pod "c5a46d67-e91b-469d-98ab-4df863a85c1e" (UID: "c5a46d67-e91b-469d-98ab-4df863a85c1e"). InnerVolumeSpecName "kube-api-access-srppl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:28:33.985674 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.985647 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5a46d67-e91b-469d-98ab-4df863a85c1e" containerID="9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289" exitCode=0 Jul 27 23:28:33.985772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.985689 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-psj9h" Jul 27 23:28:33.985772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.985702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-psj9h" event={"ID":"c5a46d67-e91b-469d-98ab-4df863a85c1e","Type":"ContainerDied","Data":"9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289"} Jul 27 23:28:33.985772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.985746 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-psj9h" event={"ID":"c5a46d67-e91b-469d-98ab-4df863a85c1e","Type":"ContainerDied","Data":"2db41e9b4aac423c845ad2e1c90ec7c49b837c5a7281f96f958f3ddc93078e2b"} Jul 27 23:28:33.985772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.985768 2564 scope.go:117] "RemoveContainer" containerID="9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289" Jul 27 23:28:33.994599 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.994566 2564 scope.go:117] "RemoveContainer" containerID="9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289" Jul 27 23:28:33.994929 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:28:33.994909 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289\": container with ID starting with 9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289 not found: ID does not exist" containerID="9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289" Jul 27 23:28:33.995002 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:33.994938 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289"} err="failed to get container status \"9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289\": rpc error: code = NotFound desc = could not find container \"9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289\": container with ID starting with 9620586c9fc9460ddfb767dd97fa9b8d0973d3299d8596ee4b422b1ca994e289 not found: ID does not exist" Jul 27 23:28:34.005580 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:34.005357 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-psj9h"] Jul 27 23:28:34.007535 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:34.007517 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-psj9h"] Jul 27 23:28:34.047344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:34.047322 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srppl\" (UniqueName: \"kubernetes.io/projected/c5a46d67-e91b-469d-98ab-4df863a85c1e-kube-api-access-srppl\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:28:35.782102 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:35.782072 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a46d67-e91b-469d-98ab-4df863a85c1e" path="/var/lib/kubelet/pods/c5a46d67-e91b-469d-98ab-4df863a85c1e/volumes" Jul 27 23:28:37.997150 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:37.997114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:38.052329 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:38.052265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:38.204749 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:38.204722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:41.420376 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:41.420343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:28:41.524587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:28:41.524554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:10.848465 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:10.848433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:10.948267 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:10.948209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:11.539422 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:11.539392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:12.007330 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:12.007299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:12.109324 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:12.109283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:12.179586 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:12.179552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:12.346887 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:12.346798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:29:22.605701 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:22.605667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:22.693654 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:22.693618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:32.672690 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:32.672659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:33.159460 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:33.159426 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:29:42.672772 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:42.672737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:43.642744 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:43.642710 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:29:51.764698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:51.764665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:29:53.148096 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:29:53.148064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:30:00.133231 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.133193 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh"] Jul 27 23:30:00.133681 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.133523 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a46d67-e91b-469d-98ab-4df863a85c1e" containerName="authorino" Jul 27 23:30:00.133681 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.133535 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a46d67-e91b-469d-98ab-4df863a85c1e" containerName="authorino" Jul 27 23:30:00.133681 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.133604 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a46d67-e91b-469d-98ab-4df863a85c1e" containerName="authorino" Jul 27 23:30:00.135660 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.135644 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" Jul 27 23:30:00.137921 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.137898 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h2bgr\"" Jul 27 23:30:00.142782 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.142757 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh"] Jul 27 23:30:00.147022 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.147004 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" Jul 27 23:30:00.283544 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.283520 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh"] Jul 27 23:30:00.286294 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:30:00.286265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87c96d7_5208_4774_bc0f_e3ddeaa4c2bb.slice/crio-f88467175f5126cb59db5dc2c2e2b9e29f94accc63be737b6c1c10e0b8030794 WatchSource:0}: Error finding container f88467175f5126cb59db5dc2c2e2b9e29f94accc63be737b6c1c10e0b8030794: Status 404 returned error can't find the container with id f88467175f5126cb59db5dc2c2e2b9e29f94accc63be737b6c1c10e0b8030794 Jul 27 23:30:00.422775 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:00.422688 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" event={"ID":"c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb","Type":"ContainerStarted","Data":"f88467175f5126cb59db5dc2c2e2b9e29f94accc63be737b6c1c10e0b8030794"} Jul 27 23:30:03.440895 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:03.440861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" event={"ID":"c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb","Type":"ContainerStarted","Data":"1c4f40be1041a2b55dd9431030f04b41de6d739f95deb9fcb6167f080992134d"} Jul 27 23:30:03.456071 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:03.456018 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" podStartSLOduration=1.218717394 podStartE2EDuration="3.45600169s" totalImagesPullingTime="2.237284296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:30:03.453616281 +0000 UTC m=+846.251580904" watchObservedRunningTime="2026-07-27 23:30:03.45600169 +0000 UTC m=+846.253966313" Jul 27 23:30:04.447554 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:04.447522 2564 generic.go:356] "Generic (PLEG): container finished" podID="c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb" containerID="1c4f40be1041a2b55dd9431030f04b41de6d739f95deb9fcb6167f080992134d" exitCode=0 Jul 27 23:30:04.447959 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:04.447560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" event={"ID":"c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb","Type":"ContainerDied","Data":"1c4f40be1041a2b55dd9431030f04b41de6d739f95deb9fcb6167f080992134d"} Jul 27 23:30:05.578662 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:05.578638 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" Jul 27 23:30:06.460742 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:06.460709 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" Jul 27 23:30:06.460987 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:06.460742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753250-czjrh" event={"ID":"c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb","Type":"ContainerDied","Data":"f88467175f5126cb59db5dc2c2e2b9e29f94accc63be737b6c1c10e0b8030794"} Jul 27 23:30:06.460987 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:06.460851 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88467175f5126cb59db5dc2c2e2b9e29f94accc63be737b6c1c10e0b8030794" Jul 27 23:30:10.466784 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:10.466746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:30:12.142147 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:12.142112 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:30:15.347982 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.347945 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr"] Jul 27 23:30:15.348358 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.348330 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb" containerName="cleanup" Jul 27 23:30:15.348358 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.348342 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb" containerName="cleanup" Jul 27 23:30:15.348429 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.348412 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb" containerName="cleanup" Jul 27 23:30:15.351719 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.351703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.354072 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.354045 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 23:30:15.354072 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.354060 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 23:30:15.354894 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.354873 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dz654\"" Jul 27 23:30:15.354976 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.354910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 27 23:30:15.360675 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.360653 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr"] Jul 27 23:30:15.447103 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.447068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:30:15.470197 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.470168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2339274-a34d-4b6f-8f73-48430893112b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.470344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.470204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.470344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.470226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.470344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.470255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6rcs\" (UniqueName: \"kubernetes.io/projected/a2339274-a34d-4b6f-8f73-48430893112b-kube-api-access-l6rcs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.470344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.470283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.470344 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.470306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.571562 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.571532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2339274-a34d-4b6f-8f73-48430893112b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.571562 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.571566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.571847 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.571589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.571847 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.571632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6rcs\" (UniqueName: \"kubernetes.io/projected/a2339274-a34d-4b6f-8f73-48430893112b-kube-api-access-l6rcs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.571847 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.571664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.571847 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.571700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.572183 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.572161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.572250 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.572190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.572250 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.572160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.574023 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.574005 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2339274-a34d-4b6f-8f73-48430893112b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.574171 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.574156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2339274-a34d-4b6f-8f73-48430893112b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.578576 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.578559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6rcs\" (UniqueName: \"kubernetes.io/projected/a2339274-a34d-4b6f-8f73-48430893112b-kube-api-access-l6rcs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4mkqr\" (UID: \"a2339274-a34d-4b6f-8f73-48430893112b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.662739 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.662643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:15.796172 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:15.795945 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr"] Jul 27 23:30:15.798378 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:30:15.798346 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2339274_a34d_4b6f_8f73_48430893112b.slice/crio-e67389bf944b24b9f6858d437b15c4c27e5f6d1a4fc6e42f8af81f5a95e25e9e WatchSource:0}: Error finding container e67389bf944b24b9f6858d437b15c4c27e5f6d1a4fc6e42f8af81f5a95e25e9e: Status 404 returned error can't find the container with id e67389bf944b24b9f6858d437b15c4c27e5f6d1a4fc6e42f8af81f5a95e25e9e Jul 27 23:30:16.515376 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:16.515343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" event={"ID":"a2339274-a34d-4b6f-8f73-48430893112b","Type":"ContainerStarted","Data":"e67389bf944b24b9f6858d437b15c4c27e5f6d1a4fc6e42f8af81f5a95e25e9e"} Jul 27 23:30:17.245088 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:17.244995 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:30:17.522457 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:17.522419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" event={"ID":"a2339274-a34d-4b6f-8f73-48430893112b","Type":"ContainerStarted","Data":"79445d43a213d13cb635c66df9723fbe07766bee3c82209f5a9f76d9a0962496"} Jul 27 23:30:17.541859 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:17.541780 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" podStartSLOduration=1.422405062 podStartE2EDuration="2.541763087s" totalImagesPullingTime="1.119358025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:30:17.541057345 +0000 UTC m=+860.339021968" watchObservedRunningTime="2026-07-27 23:30:17.541763087 +0000 UTC m=+860.339727712" Jul 27 23:30:25.663576 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:25.663541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:25.664046 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:25.663621 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:25.676047 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:25.676021 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:26.580414 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:26.580384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4mkqr" Jul 27 23:30:28.558063 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:28.558029 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:30:28.711711 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:28.711678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:30:37.467723 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:37.467681 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:30:40.484548 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:30:40.484512 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:31:28.792349 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.792314 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77fc7d94d5-vx5gf"] Jul 27 23:31:28.794718 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.794701 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:28.797942 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.797920 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 27 23:31:28.798064 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.797918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nx6j8\"" Jul 27 23:31:28.805702 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.805680 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77fc7d94d5-vx5gf"] Jul 27 23:31:28.897387 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.897364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78af2b2a-0f01-43cf-8ca2-24aee613bf41-tls-cert\") pod \"authorino-77fc7d94d5-vx5gf\" (UID: \"78af2b2a-0f01-43cf-8ca2-24aee613bf41\") " pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:28.897502 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.897397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxpf\" (UniqueName: \"kubernetes.io/projected/78af2b2a-0f01-43cf-8ca2-24aee613bf41-kube-api-access-cwxpf\") pod \"authorino-77fc7d94d5-vx5gf\" (UID: \"78af2b2a-0f01-43cf-8ca2-24aee613bf41\") " pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:28.998651 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.998623 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78af2b2a-0f01-43cf-8ca2-24aee613bf41-tls-cert\") pod \"authorino-77fc7d94d5-vx5gf\" (UID: \"78af2b2a-0f01-43cf-8ca2-24aee613bf41\") " pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:28.998789 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:28.998661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxpf\" (UniqueName: \"kubernetes.io/projected/78af2b2a-0f01-43cf-8ca2-24aee613bf41-kube-api-access-cwxpf\") pod \"authorino-77fc7d94d5-vx5gf\" (UID: \"78af2b2a-0f01-43cf-8ca2-24aee613bf41\") " pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:29.001348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.001330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/78af2b2a-0f01-43cf-8ca2-24aee613bf41-tls-cert\") pod \"authorino-77fc7d94d5-vx5gf\" (UID: \"78af2b2a-0f01-43cf-8ca2-24aee613bf41\") " pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:29.006096 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.006075 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxpf\" (UniqueName: \"kubernetes.io/projected/78af2b2a-0f01-43cf-8ca2-24aee613bf41-kube-api-access-cwxpf\") pod \"authorino-77fc7d94d5-vx5gf\" (UID: \"78af2b2a-0f01-43cf-8ca2-24aee613bf41\") " pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:29.104069 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.103995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" Jul 27 23:31:29.235666 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.235639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77fc7d94d5-vx5gf"] Jul 27 23:31:29.237004 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:31:29.236975 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78af2b2a_0f01_43cf_8ca2_24aee613bf41.slice/crio-b774ffeebc7ac29996d084e0cf0363c0876f77b0edffdd76b0b93a5391df913d WatchSource:0}: Error finding container b774ffeebc7ac29996d084e0cf0363c0876f77b0edffdd76b0b93a5391df913d: Status 404 returned error can't find the container with id b774ffeebc7ac29996d084e0cf0363c0876f77b0edffdd76b0b93a5391df913d Jul 27 23:31:29.897477 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.897389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" event={"ID":"78af2b2a-0f01-43cf-8ca2-24aee613bf41","Type":"ContainerStarted","Data":"f62bf444727728a781b19ff41d16d3864da0d3cf180d1756bf47eafaafe0a1d4"} Jul 27 23:31:29.897477 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.897434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" event={"ID":"78af2b2a-0f01-43cf-8ca2-24aee613bf41","Type":"ContainerStarted","Data":"b774ffeebc7ac29996d084e0cf0363c0876f77b0edffdd76b0b93a5391df913d"} Jul 27 23:31:29.914229 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:29.914175 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77fc7d94d5-vx5gf" podStartSLOduration=1.509316656 podStartE2EDuration="1.914157052s" totalImagesPullingTime="404.840396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:31:29.912222737 +0000 UTC m=+932.710187362" watchObservedRunningTime="2026-07-27 23:31:29.914157052 +0000 UTC m=+932.712121675" Jul 27 23:31:30.415985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:30.415951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:31:30.478109 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:30.478073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:31:31.505690 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:31.505640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:31:32.565737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:31:32.565705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:32:00.247507 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:32:00.247469 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:32:09.747055 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:32:09.747015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:32:19.250475 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:32:19.250394 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:32:30.867985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:32:30.867951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:32:30.965967 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:32:30.965935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:32:46.514446 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:32:46.514411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:33:08.727218 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:33:08.727183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:33:35.262378 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:33:35.262341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:33:36.492381 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:33:36.492340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:03.642800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:03.642719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:34:12.684828 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:12.684781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:12.785830 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:12.785794 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:12.904221 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:12.904190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:13.347569 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:13.347527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:34:18.524669 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:18.524634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:18.590273 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:18.590239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:18.659842 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:18.659790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:18.851478 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:18.851391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:24.598085 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:24.598056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:24.656862 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:24.656806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:36.210532 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:36.210499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:36.262176 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:36.262143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:34:41.750110 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:41.750066 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:34:51.439429 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:34:51.439396 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:35:02.143982 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:35:02.143948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:35:09.938302 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:35:09.938256 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:35:20.952746 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:35:20.952648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:35:33.538968 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:35:33.538932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:35:41.042224 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:35:41.042185 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:35:50.445980 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:35:50.445947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:36:15.247383 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:36:15.247343 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:36:24.845128 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:36:24.845092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:36:48.552307 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:36:48.552271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:36:59.539670 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:36:59.539628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:37:10.441828 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:37:10.441781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:37:25.743694 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:37:25.743648 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:37:36.751773 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:37:36.751723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:37:46.538363 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:37:46.538323 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:37:56.142246 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:37:56.142203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:38:05.636510 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:38:05.636465 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:38:10.466710 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:38:10.466676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:38:26.760011 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:38:26.759964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:35.126808 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:39:35.126772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:39:52.549508 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:39:52.549429 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:40:02.744987 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:40:02.744955 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:40:17.445212 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:40:17.445168 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:40:29.444303 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:40:29.444260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:40:39.342581 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:40:39.342549 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:40:49.247027 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:40:49.246994 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:41:00.252948 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:41:00.252912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:41:10.250123 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:41:10.250086 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:41:20.641811 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:41:20.641729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:41:30.243884 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:41:30.243844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:41:43.144124 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:41:43.144089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:42:02.844777 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:02.844743 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:42:12.639351 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:12.639316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:42:14.180067 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:14.180028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:17.754754 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:17.754713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:25.638986 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:25.638949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:42:45.152697 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:45.152662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:46.843069 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:46.843035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:42:48.346435 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:42:48.346399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:43:01.045498 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:43:01.045461 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:43:10.644612 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:43:10.644576 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:43:23.850708 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:43:23.850670 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:43:38.445996 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:43:38.445963 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:43:48.064217 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:43:48.064171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:43:59.745027 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:43:59.744991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:44:13.247839 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:44:13.247742 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:44:25.042968 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:44:25.042937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:44:35.142545 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:44:35.142506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:44:43.789088 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:44:43.789047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:44:45.644947 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:44:45.644910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:44:56.244553 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:44:56.244522 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:45:16.649945 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:45:16.649913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:45:28.952452 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:45:28.952415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:45:41.050493 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:45:41.050456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:45:50.245738 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:45:50.245664 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:45:53.822245 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:45:53.822210 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:00.279241 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:00.279199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:46:11.742208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:11.742169 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:46:17.372418 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:17.372382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:20.413375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:20.413333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:20.512971 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:20.512933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:20.847389 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:20.847362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:21.851203 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:21.851166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:46:26.468990 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:26.468950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:26.570014 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:26.569965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:26.677921 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:26.677888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:26.965584 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:26.965551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:27.286459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:27.286427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:29.341830 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:29.341780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:29.414965 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:29.414932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:29.482972 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:29.482947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:29.581480 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:29.581447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:29.991614 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:29.991580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:30.074390 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:30.074362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:46:33.739289 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:46:33.739254 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:47:05.372932 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:05.372895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:05.432022 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:05.431977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:05.487783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:05.487747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:08.637033 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:08.637000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:47:09.983439 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:09.983402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:10.154250 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:10.154222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:10.383776 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:10.383738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:10.638193 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:10.638123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:10.735646 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:10.735617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:10.849893 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:10.849843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:11.164278 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:11.164243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:11.357867 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:11.357838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:16.110377 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:16.110329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:16.218886 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:16.218839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:16.517291 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:16.517246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:16.783702 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:16.783612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:16.877597 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:16.877563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:19.548282 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:19.548209 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:47:51.496790 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:51.496751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:51.567857 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:51.567797 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:51.648722 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:51.648689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:54.443118 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:54.443081 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:47:56.190317 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:56.190278 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:56.338060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:56.338030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:56.570087 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:56.570053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:56.782711 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:56.782671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:57.080309 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:57.080273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:57.812518 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:57.812485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:59.702587 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:59.702538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:59.789979 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:59.789944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:47:59.852223 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:47:59.852186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:00.264431 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:00.264397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:00.362041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:00.361995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:00.432285 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:00.432243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:05.847086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:05.847051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:48:21.644438 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:21.644406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:35.853958 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:35.853921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:35.953651 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:35.953605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:36.054657 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:36.054621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:38.845198 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:38.845166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:48:40.544744 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:40.544706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:41.699208 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:41.699165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:41.926686 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:41.926648 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:42.134448 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:42.134416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:42.423981 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:42.423907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:42.894840 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:42.894774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:44.486664 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:44.486633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:44.566412 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:44.566380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:44.634327 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:44.634287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:44.908226 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:44.908193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:45.248707 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:45.248682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:45.353860 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:45.353798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:48:50.543592 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:48:50.543510 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:49:19.731099 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:19.731059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:19.793308 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:19.793273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:19.893676 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:19.893644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:22.643574 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:22.643542 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:49:24.412931 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:24.412891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:24.566962 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:24.566928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:24.795797 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:24.795757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:25.253288 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:25.253253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:25.352537 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:25.352498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:25.453294 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:25.453263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:26.641117 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:26.641066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:26.808617 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:26.808582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:32.545909 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:32.545868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:32.655439 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:32.655407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:32.750506 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:32.750451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:32.847674 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:32.847581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:33.314967 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:33.314924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:49:59.208882 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:49:59.208844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:05.597916 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:05.597871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:05.703175 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:05.703131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:05.799703 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:05.799666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:21.635341 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:21.635306 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:27.173123 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:27.173087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:27.228557 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:27.228526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:27.303702 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:27.303663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:27.404868 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:27.404839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:37.849799 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:37.849758 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:38.616179 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:38.616146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:44.838541 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:44.838492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:44.941258 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:44.941221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:45.040118 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:45.040073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:45.140569 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:45.140472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:48.253186 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:48.253147 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:50:56.506986 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:56.506947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:56.605183 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:56.605148 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:56.708167 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:56.708139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:50:58.331630 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:50:58.331594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:00.137643 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:00.137605 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:51:04.078985 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:04.078945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:04.137392 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:04.137355 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:04.744094 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:04.744063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:09.942248 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:09.942213 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:51:10.330486 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:10.330453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:10.388233 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:10.388196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:10.549363 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:10.549318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:10.840906 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:10.840872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:10.967002 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:10.966967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:13.851738 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:13.851689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:13.925571 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:13.925536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:13.990267 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:13.990222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:14.262713 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:14.262678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:14.620016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:14.619920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:14.720588 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:14.720551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:18.318742 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:18.318704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:18.932511 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:18.932476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:19.029801 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:19.029760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:19.337633 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:19.337599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:20.040768 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:20.040732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:51:25.111357 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:25.111321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:25.168394 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:25.168354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:35.989373 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:35.989334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:36.054254 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:36.054223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:36.110958 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:36.110921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:38.741409 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:38.741375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:51:40.516455 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:40.516412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:40.876728 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:40.876650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:46.629681 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:46.629645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:46.679292 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:46.679259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:46.889274 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:46.889182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:47.107688 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:47.107658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:47.231348 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:47.231258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:47.324016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:47.323980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:47.436657 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:47.436618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:53.108834 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:53.108732 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:53.209425 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:53.209391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:53.309401 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:53.309366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:53.438060 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:53.437967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:58.924459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:58.924431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:51:59.026765 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:51:59.026737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:05.657205 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:05.657175 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:05.937670 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:05.937579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:06.165540 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:06.165511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:09.098158 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:09.098129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:09.212391 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:09.212359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:09.304541 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:09.304504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:09.398312 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:09.398223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:09.759661 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:09.759632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:09.851548 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:09.851509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:13.622706 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:13.622668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:14.342544 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:14.342504 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:52:17.056263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:17.056225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:21.896024 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:21.895981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:23.466402 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:23.466371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:52:27.344917 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:27.344882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:52:48.545235 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:52:48.545197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:53:07.649309 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:07.649275 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:53:09.213766 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:09.213732 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:09.279087 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:09.279052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:09.335238 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:09.335205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:13.717258 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:13.717209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:16.782384 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:16.782351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:17.442799 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:17.442764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:53:21.861511 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:21.861478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:32.326600 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:32.326563 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:47.182838 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:47.182787 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:47.232997 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:47.232962 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:57.901645 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:57.901610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:53:58.838799 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:53:58.838762 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:04.242923 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:04.242890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:04.307252 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:04.307222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:04.514858 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:04.514798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:04.730830 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:04.730793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:05.739049 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:05.739018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:05.835583 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:05.835541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:05.935089 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:05.935051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:06.629098 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:06.629062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:06.729624 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:06.729596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:06.830441 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:06.830412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:08.119553 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:08.119511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:08.550880 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:08.550842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:54:13.951503 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:13.951464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:14.004156 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:14.004131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:14.603800 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:14.603762 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:20.149201 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:20.149170 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:54:20.560100 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:20.560050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:20.664765 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:20.664705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:20.761007 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:20.760961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:31.244127 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:31.244094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:31.554287 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:31.554237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:37.901178 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:37.901138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:52.478358 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:52.478279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:58.708192 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:58.708157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:58.809021 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:58.808987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:59.026783 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:59.026754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:59.318053 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:59.317974 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:54:59.452396 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:54:59.452346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:02.045478 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:02.045437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:03.328244 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:03.328198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:06.141623 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:06.141590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:55:06.509649 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:06.509598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:06.915713 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:06.915627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:07.152153 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:07.152099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:07.481374 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:07.481339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:10.457163 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:10.457124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:10.550898 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:10.550865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:10.615688 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:10.615652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:10.716273 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:10.716194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:11.269508 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:11.269458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:11.363363 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:11.363330 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:14.976322 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:14.976292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:22.850402 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:22.850366 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:55:35.466502 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:35.466462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:55:40.148664 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:55:40.148633 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:56:08.148038 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.147997 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r"] Jul 27 23:56:08.151502 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.151486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.155988 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.155796 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3ed4f0\"/\"default-dockercfg-pxwt8\"" Jul 27 23:56:08.155988 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.155854 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3ed4f0\"/\"test-model-3ed4f0-kserve-self-signed-certs\"" Jul 27 23:56:08.155988 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.155902 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3ed4f0\"/\"kube-root-ca.crt\"" Jul 27 23:56:08.155988 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.155967 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3ed4f0\"/\"openshift-service-ca.crt\"" Jul 27 23:56:08.159624 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.159605 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r"] Jul 27 23:56:08.185500 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.185475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:56:08.224037 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.224014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-model-cache\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.224140 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.224054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-home\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.224140 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.224073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tls-certs\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.224211 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.224139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-dshm\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.224211 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.224182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tmp-dir\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.224276 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.224210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-998rm\" (UniqueName: \"kubernetes.io/projected/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-kube-api-access-998rm\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.324579 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.324558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-dshm\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.324703 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.324595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tmp-dir\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.324703 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.324626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-998rm\" (UniqueName: \"kubernetes.io/projected/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-kube-api-access-998rm\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.324703 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.324685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-model-cache\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.324911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.324733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-home\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.324911 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.324757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tls-certs\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.325095 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.325068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tmp-dir\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.325351 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.325329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-home\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.325445 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.325329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-model-cache\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.327278 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.327258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-dshm\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.327621 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.327599 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tls-certs\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.332683 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.332665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-998rm\" (UniqueName: \"kubernetes.io/projected/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-kube-api-access-998rm\") pod \"test-model-3ed4f0-kserve-749645bb4f-r654r\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.464327 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.464261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:08.594966 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.594941 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r"] Jul 27 23:56:08.596877 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:56:08.596846 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12cd99db_afe8_4d19_80c8_cafbdeddc8c9.slice/crio-072109f5233645b0012203f067b957e4fcbd3efaf4281f04f8a7bb9c889afecc WatchSource:0}: Error finding container 072109f5233645b0012203f067b957e4fcbd3efaf4281f04f8a7bb9c889afecc: Status 404 returned error can't find the container with id 072109f5233645b0012203f067b957e4fcbd3efaf4281f04f8a7bb9c889afecc Jul 27 23:56:08.651619 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:08.651593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" event={"ID":"12cd99db-afe8-4d19-80c8-cafbdeddc8c9","Type":"ContainerStarted","Data":"072109f5233645b0012203f067b957e4fcbd3efaf4281f04f8a7bb9c889afecc"} Jul 27 23:56:09.658240 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:09.658205 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" event={"ID":"12cd99db-afe8-4d19-80c8-cafbdeddc8c9","Type":"ContainerStarted","Data":"4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b"} Jul 27 23:56:09.674517 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:09.674471 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" podStartSLOduration=1.6744577010000001 podStartE2EDuration="1.674457701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:56:09.671744561 +0000 UTC m=+2412.469709185" watchObservedRunningTime="2026-07-27 23:56:09.674457701 +0000 UTC m=+2412.472422326" Jul 27 23:56:12.939758 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:12.939711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:56:18.464375 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:18.464342 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:18.464797 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:18.464394 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:18.476762 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:18.476740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:18.714886 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:18.714767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:56:27.742973 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:27.742939 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:56:50.443794 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:50.443738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:56:51.623290 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:51.623257 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r"] Jul 27 23:56:51.643357 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:51.643323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:56:56.035074 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:56:56.035036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:57:02.187705 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:02.187665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:02.299913 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:02.299872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:06.643187 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.643126 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" podUID="12cd99db-afe8-4d19-80c8-cafbdeddc8c9" containerName="main" containerID="cri-o://4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b" gracePeriod=45 Jul 27 23:57:06.809009 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.808975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:06.902314 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.902253 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:57:06.953279 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.953249 2564 generic.go:356] "Generic (PLEG): container finished" podID="12cd99db-afe8-4d19-80c8-cafbdeddc8c9" containerID="4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b" exitCode=0 Jul 27 23:57:06.953459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.953287 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" Jul 27 23:57:06.953459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.953310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" event={"ID":"12cd99db-afe8-4d19-80c8-cafbdeddc8c9","Type":"ContainerDied","Data":"4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b"} Jul 27 23:57:06.953459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.953349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r" event={"ID":"12cd99db-afe8-4d19-80c8-cafbdeddc8c9","Type":"ContainerDied","Data":"072109f5233645b0012203f067b957e4fcbd3efaf4281f04f8a7bb9c889afecc"} Jul 27 23:57:06.953459 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.953373 2564 scope.go:117] "RemoveContainer" containerID="4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b" Jul 27 23:57:06.964192 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.964153 2564 scope.go:117] "RemoveContainer" containerID="4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b" Jul 27 23:57:06.964566 ip-10-0-141-248 kubenswrapper[2564]: E0727 23:57:06.964545 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b\": container with ID starting with 4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b not found: ID does not exist" containerID="4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b" Jul 27 23:57:06.964635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:06.964581 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b"} err="failed to get container status \"4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b\": rpc error: code = NotFound desc = could not find container \"4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b\": container with ID starting with 4287ad7685c9f8df61d59deb511dc4a9bbb5f9f664a7df1cd8d59c27c674f24b not found: ID does not exist" Jul 27 23:57:07.028801 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.028771 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tmp-dir\") pod \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " Jul 27 23:57:07.028801 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.028805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-model-cache\") pod \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " Jul 27 23:57:07.029043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.028874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-home\") pod \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " Jul 27 23:57:07.029043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.028936 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tls-certs\") pod \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " Jul 27 23:57:07.029043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.028956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-998rm\" (UniqueName: \"kubernetes.io/projected/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-kube-api-access-998rm\") pod \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " Jul 27 23:57:07.029043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.028992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-dshm\") pod \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\" (UID: \"12cd99db-afe8-4d19-80c8-cafbdeddc8c9\") " Jul 27 23:57:07.029043 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.029017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12cd99db-afe8-4d19-80c8-cafbdeddc8c9" (UID: "12cd99db-afe8-4d19-80c8-cafbdeddc8c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:57:07.029279 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.029165 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-home" (OuterVolumeSpecName: "home") pod "12cd99db-afe8-4d19-80c8-cafbdeddc8c9" (UID: "12cd99db-afe8-4d19-80c8-cafbdeddc8c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:57:07.029279 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.029176 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tmp-dir\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:57:07.029279 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.029195 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "12cd99db-afe8-4d19-80c8-cafbdeddc8c9" (UID: "12cd99db-afe8-4d19-80c8-cafbdeddc8c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:57:07.031267 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.031246 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-dshm" (OuterVolumeSpecName: "dshm") pod "12cd99db-afe8-4d19-80c8-cafbdeddc8c9" (UID: "12cd99db-afe8-4d19-80c8-cafbdeddc8c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 23:57:07.031325 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.031308 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-kube-api-access-998rm" (OuterVolumeSpecName: "kube-api-access-998rm") pod "12cd99db-afe8-4d19-80c8-cafbdeddc8c9" (UID: "12cd99db-afe8-4d19-80c8-cafbdeddc8c9"). InnerVolumeSpecName "kube-api-access-998rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 23:57:07.031369 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.031346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12cd99db-afe8-4d19-80c8-cafbdeddc8c9" (UID: "12cd99db-afe8-4d19-80c8-cafbdeddc8c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 23:57:07.130428 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.130399 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-model-cache\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:57:07.130428 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.130426 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-home\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:57:07.130428 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.130435 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-tls-certs\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:57:07.130629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.130443 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-998rm\" (UniqueName: \"kubernetes.io/projected/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-kube-api-access-998rm\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:57:07.130629 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.130452 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd99db-afe8-4d19-80c8-cafbdeddc8c9-dshm\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 27 23:57:07.174191 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.174116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:07.273020 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.272989 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r"] Jul 27 23:57:07.277929 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.277902 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3ed4f0/test-model-3ed4f0-kserve-749645bb4f-r654r"] Jul 27 23:57:07.783047 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:07.783019 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cd99db-afe8-4d19-80c8-cafbdeddc8c9" path="/var/lib/kubelet/pods/12cd99db-afe8-4d19-80c8-cafbdeddc8c9/volumes" Jul 27 23:57:12.245093 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:12.245053 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:57:12.929298 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:12.929260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:12.985803 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:12.985768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:13.187489 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:13.187409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:13.386359 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:13.386323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:24.073753 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:24.073712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:24.137263 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:24.137230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:24.193469 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:24.193432 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:27.439461 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:27.439430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:57:28.565603 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:28.565570 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:28.920075 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:28.919986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:34.699215 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:34.699177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:34.755176 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:34.755132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:34.945748 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:34.945719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:35.162026 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:35.161987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:35.478013 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:35.477924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:35.782110 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:35.782047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:35.998504 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:35.998456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:38.861547 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:38.861518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:38.931057 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:38.931021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:39.041954 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:39.041911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:39.179995 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:39.179912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:39.615160 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:39.615123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:39.704698 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:39.704664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:43.336487 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:43.336445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:43.781592 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:43.781552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:44.066550 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:44.066444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:44.234041 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:44.234007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:46.551892 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:46.551856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:57:47.306068 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:47.306032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:47.406127 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:47.406086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:47.534769 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:47.534732 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:48.076264 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:48.076224 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:48.174034 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:48.174000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:48.259446 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:48.259416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:57:51.788916 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:57:51.788845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:03.551669 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:03.551624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:58:15.331458 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:15.331423 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:15.398542 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:15.398505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:15.449144 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:15.449115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:19.777429 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:19.777393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:20.128876 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:20.128777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:20.939335 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:20.939298 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:58:25.704550 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:25.704513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:25.758666 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:25.758631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:25.962696 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:25.962610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:26.176999 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:26.176951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:33.852984 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:33.852950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:41.692674 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:41.692640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:41.793898 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:41.793855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:43.741423 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:43.741390 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:58:46.348120 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:46.348081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:46.696352 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:46.696258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:52.551570 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:52.551537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:52.652097 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:52.652070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:52.816134 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:52.816050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:53.038607 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:53.038571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:53.363898 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:53.363862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:53.658442 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:53.658353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:53.885288 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:53.885254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:56.338214 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:56.338176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:56.431474 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:56.431440 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:56.556667 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:56.556635 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:56.724679 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:56.724595 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:57.057038 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:57.057008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:58:57.159879 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:58:57.159848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:00.805645 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:00.805607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:01.129737 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:01.129657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:01.428399 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:01.428316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:01.853551 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:01.853516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:04.713193 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:04.713155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:04.788137 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:04.788107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:04.851939 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:04.851905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:04.994172 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:04.994097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:05.276785 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:05.276751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:05.372087 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:05.372054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:05.838510 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:05.838477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:59:09.198926 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:09.198880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:16.980225 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:16.980191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:19.850071 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:19.849991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 27 23:59:25.130600 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:25.130565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:25.179433 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:25.179402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:53.040460 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.040427 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx"] Jul 27 23:59:53.041120 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.041098 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cd99db-afe8-4d19-80c8-cafbdeddc8c9" containerName="main" Jul 27 23:59:53.041211 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.041122 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cd99db-afe8-4d19-80c8-cafbdeddc8c9" containerName="main" Jul 27 23:59:53.041272 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.041228 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cd99db-afe8-4d19-80c8-cafbdeddc8c9" containerName="main" Jul 27 23:59:53.044155 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.044136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.047515 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.047495 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"kube-root-ca.crt\"" Jul 27 23:59:53.047635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.047568 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"auth-test-model-4a8122-kserve-self-signed-certs\"" Jul 27 23:59:53.047635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.047582 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"default-dockercfg-gdmj9\"" Jul 27 23:59:53.047635 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.047535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"openshift-service-ca.crt\"" Jul 27 23:59:53.053177 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.052430 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx"] Jul 27 23:59:53.081480 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.081451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 27 23:59:53.083851 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.083802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjhv\" (UniqueName: \"kubernetes.io/projected/a317b73f-bb82-4931-bea0-67c67eea76fa-kube-api-access-ctjhv\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.083949 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.083932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-tmp-dir\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.084016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.083963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a317b73f-bb82-4931-bea0-67c67eea76fa-tls-certs\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.084016 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.083995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-home\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.084086 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.084046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-dshm\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.084127 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.084082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-model-cache\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.184612 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.184586 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-dshm\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.184715 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.184631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-model-cache\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.184715 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.184668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjhv\" (UniqueName: \"kubernetes.io/projected/a317b73f-bb82-4931-bea0-67c67eea76fa-kube-api-access-ctjhv\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.184861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.184737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-tmp-dir\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.184861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.184762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a317b73f-bb82-4931-bea0-67c67eea76fa-tls-certs\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.184861 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.184790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-home\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.185075 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.185055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-model-cache\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.185138 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.185094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-tmp-dir\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.185192 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.185145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-home\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.186939 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.186921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-dshm\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.187215 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.187197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a317b73f-bb82-4931-bea0-67c67eea76fa-tls-certs\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.192418 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.192398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjhv\" (UniqueName: \"kubernetes.io/projected/a317b73f-bb82-4931-bea0-67c67eea76fa-kube-api-access-ctjhv\") pod \"auth-test-model-4a8122-kserve-5466c574b7-zwzzx\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.355860 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.355777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 27 23:59:53.480951 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.480925 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx"] Jul 27 23:59:53.482586 ip-10-0-141-248 kubenswrapper[2564]: W0727 23:59:53.482560 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda317b73f_bb82_4931_bea0_67c67eea76fa.slice/crio-c677dfeb20658491188a0850c3e0fdc3dff21480bbb9fbb3aa59647d72640262 WatchSource:0}: Error finding container c677dfeb20658491188a0850c3e0fdc3dff21480bbb9fbb3aa59647d72640262: Status 404 returned error can't find the container with id c677dfeb20658491188a0850c3e0fdc3dff21480bbb9fbb3aa59647d72640262 Jul 27 23:59:53.816210 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.816178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" event={"ID":"a317b73f-bb82-4931-bea0-67c67eea76fa","Type":"ContainerStarted","Data":"a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038"} Jul 27 23:59:53.816382 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.816218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" event={"ID":"a317b73f-bb82-4931-bea0-67c67eea76fa","Type":"ContainerStarted","Data":"c677dfeb20658491188a0850c3e0fdc3dff21480bbb9fbb3aa59647d72640262"} Jul 27 23:59:53.834021 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:53.833968 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" podStartSLOduration=0.833941612 podStartE2EDuration="833.941612ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 23:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 23:59:53.831655549 +0000 UTC m=+2636.629620173" watchObservedRunningTime="2026-07-27 23:59:53.833941612 +0000 UTC m=+2636.631906235" Jul 27 23:59:57.443909 ip-10-0-141-248 kubenswrapper[2564]: I0727 23:59:57.443874 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:00:00.131873 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.131832 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29753280-skcj7"] Jul 28 00:00:00.135792 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.135770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.137916 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.137892 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 28 00:00:00.138064 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.138048 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-6hnkb\"" Jul 28 00:00:00.145360 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.145327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29753280-skcj7"] Jul 28 00:00:00.241385 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.241344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5v4t\" (UniqueName: \"kubernetes.io/projected/24bcd7aa-9300-441b-b872-f5f5013abd0f-kube-api-access-k5v4t\") pod \"image-pruner-29753280-skcj7\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.241385 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.241395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24bcd7aa-9300-441b-b872-f5f5013abd0f-serviceca\") pod \"image-pruner-29753280-skcj7\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.342228 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.342190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5v4t\" (UniqueName: \"kubernetes.io/projected/24bcd7aa-9300-441b-b872-f5f5013abd0f-kube-api-access-k5v4t\") pod \"image-pruner-29753280-skcj7\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.342415 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.342241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24bcd7aa-9300-441b-b872-f5f5013abd0f-serviceca\") pod \"image-pruner-29753280-skcj7\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.342907 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.342884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24bcd7aa-9300-441b-b872-f5f5013abd0f-serviceca\") pod \"image-pruner-29753280-skcj7\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.349992 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.349971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5v4t\" (UniqueName: \"kubernetes.io/projected/24bcd7aa-9300-441b-b872-f5f5013abd0f-kube-api-access-k5v4t\") pod \"image-pruner-29753280-skcj7\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.461880 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.461767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:00.594246 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.594218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29753280-skcj7"] Jul 28 00:00:00.595473 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:00:00.595446 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bcd7aa_9300_441b_b872_f5f5013abd0f.slice/crio-3da58d52e5ef89487e1845a75d4d4954ec52ca9ba2916db5f13362b8f59d5eaa WatchSource:0}: Error finding container 3da58d52e5ef89487e1845a75d4d4954ec52ca9ba2916db5f13362b8f59d5eaa: Status 404 returned error can't find the container with id 3da58d52e5ef89487e1845a75d4d4954ec52ca9ba2916db5f13362b8f59d5eaa Jul 28 00:00:00.853418 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.853381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-skcj7" event={"ID":"24bcd7aa-9300-441b-b872-f5f5013abd0f","Type":"ContainerStarted","Data":"1cd5dc0145d8ad08ba921a93417294ec96a40b40cc3342d7b0a916ed0ea2bdc1"} Jul 28 00:00:00.853606 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.853439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-skcj7" event={"ID":"24bcd7aa-9300-441b-b872-f5f5013abd0f","Type":"ContainerStarted","Data":"3da58d52e5ef89487e1845a75d4d4954ec52ca9ba2916db5f13362b8f59d5eaa"} Jul 28 00:00:00.868795 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:00.868740 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29753280-skcj7" podStartSLOduration=0.868723974 podStartE2EDuration="868.723974ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:00:00.866626826 +0000 UTC m=+2643.664591461" watchObservedRunningTime="2026-07-28 00:00:00.868723974 +0000 UTC m=+2643.666688599" Jul 28 00:00:01.860420 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:01.860389 2564 generic.go:356] "Generic (PLEG): container finished" podID="24bcd7aa-9300-441b-b872-f5f5013abd0f" containerID="1cd5dc0145d8ad08ba921a93417294ec96a40b40cc3342d7b0a916ed0ea2bdc1" exitCode=0 Jul 28 00:00:01.860797 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:01.860440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-skcj7" event={"ID":"24bcd7aa-9300-441b-b872-f5f5013abd0f","Type":"ContainerDied","Data":"1cd5dc0145d8ad08ba921a93417294ec96a40b40cc3342d7b0a916ed0ea2bdc1"} Jul 28 00:00:03.000834 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.000793 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:03.066455 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.066430 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5v4t\" (UniqueName: \"kubernetes.io/projected/24bcd7aa-9300-441b-b872-f5f5013abd0f-kube-api-access-k5v4t\") pod \"24bcd7aa-9300-441b-b872-f5f5013abd0f\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " Jul 28 00:00:03.066594 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.066462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24bcd7aa-9300-441b-b872-f5f5013abd0f-serviceca\") pod \"24bcd7aa-9300-441b-b872-f5f5013abd0f\" (UID: \"24bcd7aa-9300-441b-b872-f5f5013abd0f\") " Jul 28 00:00:03.066796 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.066774 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bcd7aa-9300-441b-b872-f5f5013abd0f-serviceca" (OuterVolumeSpecName: "serviceca") pod "24bcd7aa-9300-441b-b872-f5f5013abd0f" (UID: "24bcd7aa-9300-441b-b872-f5f5013abd0f"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:00:03.068572 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.068553 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bcd7aa-9300-441b-b872-f5f5013abd0f-kube-api-access-k5v4t" (OuterVolumeSpecName: "kube-api-access-k5v4t") pod "24bcd7aa-9300-441b-b872-f5f5013abd0f" (UID: "24bcd7aa-9300-441b-b872-f5f5013abd0f"). InnerVolumeSpecName "kube-api-access-k5v4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:00:03.167050 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.166991 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5v4t\" (UniqueName: \"kubernetes.io/projected/24bcd7aa-9300-441b-b872-f5f5013abd0f-kube-api-access-k5v4t\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:00:03.167050 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.167015 2564 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24bcd7aa-9300-441b-b872-f5f5013abd0f-serviceca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:00:03.356484 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.356459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 28 00:00:03.356623 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.356503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 28 00:00:03.368156 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.368134 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 28 00:00:03.872280 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.872251 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29753280-skcj7" Jul 28 00:00:03.872280 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.872264 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29753280-skcj7" event={"ID":"24bcd7aa-9300-441b-b872-f5f5013abd0f","Type":"ContainerDied","Data":"3da58d52e5ef89487e1845a75d4d4954ec52ca9ba2916db5f13362b8f59d5eaa"} Jul 28 00:00:03.872503 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.872303 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da58d52e5ef89487e1845a75d4d4954ec52ca9ba2916db5f13362b8f59d5eaa" Jul 28 00:00:03.883923 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:03.883902 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 28 00:00:19.746728 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:19.746688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:00:21.081117 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:21.081080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:00:21.447431 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:21.447350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:00:48.837902 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:00:48.837830 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:01:07.748238 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:01:07.748196 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:01:23.017044 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:01:23.017001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:01:29.850486 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:01:29.850449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:01:39.885589 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:01:39.885549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:01:49.045428 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:01:49.045384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:02:10.247846 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:02:10.247794 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:02:23.059353 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:02:23.059321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:02:24.842003 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:02:24.841971 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:02:36.645333 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:02:36.645297 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:02:52.441844 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:02:52.441798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:03:06.952340 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:03:06.952303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:03:19.142895 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:03:19.142858 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:03:34.237975 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:03:34.237941 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:03:52.649128 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:03:52.649045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:04:14.844420 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:14.844384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:04:17.745192 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:17.745158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:18.624634 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:18.624606 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx"] Jul 28 00:04:18.643663 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:18.643628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:30.668922 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:30.668874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:31.445511 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:31.445477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:04:33.644119 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:33.644075 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" podUID="a317b73f-bb82-4931-bea0-67c67eea76fa" containerName="main" containerID="cri-o://a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038" gracePeriod=45 Jul 28 00:04:33.901276 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:33.901215 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 28 00:04:34.004549 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004504 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a317b73f-bb82-4931-bea0-67c67eea76fa-tls-certs\") pod \"a317b73f-bb82-4931-bea0-67c67eea76fa\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " Jul 28 00:04:34.004736 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-model-cache\") pod \"a317b73f-bb82-4931-bea0-67c67eea76fa\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " Jul 28 00:04:34.004736 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004660 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctjhv\" (UniqueName: \"kubernetes.io/projected/a317b73f-bb82-4931-bea0-67c67eea76fa-kube-api-access-ctjhv\") pod \"a317b73f-bb82-4931-bea0-67c67eea76fa\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " Jul 28 00:04:34.004736 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-dshm\") pod \"a317b73f-bb82-4931-bea0-67c67eea76fa\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " Jul 28 00:04:34.004908 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004750 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-home\") pod \"a317b73f-bb82-4931-bea0-67c67eea76fa\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " Jul 28 00:04:34.004908 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004789 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-tmp-dir\") pod \"a317b73f-bb82-4931-bea0-67c67eea76fa\" (UID: \"a317b73f-bb82-4931-bea0-67c67eea76fa\") " Jul 28 00:04:34.004908 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.004860 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "a317b73f-bb82-4931-bea0-67c67eea76fa" (UID: "a317b73f-bb82-4931-bea0-67c67eea76fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:04:34.005115 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.005089 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-home" (OuterVolumeSpecName: "home") pod "a317b73f-bb82-4931-bea0-67c67eea76fa" (UID: "a317b73f-bb82-4931-bea0-67c67eea76fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:04:34.005182 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.005114 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-model-cache\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:04:34.005217 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.005179 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a317b73f-bb82-4931-bea0-67c67eea76fa" (UID: "a317b73f-bb82-4931-bea0-67c67eea76fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:04:34.007048 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.007021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-dshm" (OuterVolumeSpecName: "dshm") pod "a317b73f-bb82-4931-bea0-67c67eea76fa" (UID: "a317b73f-bb82-4931-bea0-67c67eea76fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:04:34.007048 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.007040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a317b73f-bb82-4931-bea0-67c67eea76fa-kube-api-access-ctjhv" (OuterVolumeSpecName: "kube-api-access-ctjhv") pod "a317b73f-bb82-4931-bea0-67c67eea76fa" (UID: "a317b73f-bb82-4931-bea0-67c67eea76fa"). InnerVolumeSpecName "kube-api-access-ctjhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:04:34.007184 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.007078 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a317b73f-bb82-4931-bea0-67c67eea76fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a317b73f-bb82-4931-bea0-67c67eea76fa" (UID: "a317b73f-bb82-4931-bea0-67c67eea76fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:04:34.106410 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.106381 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-dshm\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:04:34.106410 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.106407 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-home\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:04:34.106602 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.106418 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a317b73f-bb82-4931-bea0-67c67eea76fa-tmp-dir\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:04:34.106602 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.106431 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a317b73f-bb82-4931-bea0-67c67eea76fa-tls-certs\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:04:34.106602 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.106443 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctjhv\" (UniqueName: \"kubernetes.io/projected/a317b73f-bb82-4931-bea0-67c67eea76fa-kube-api-access-ctjhv\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:04:34.358806 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.358770 2564 generic.go:356] "Generic (PLEG): container finished" podID="a317b73f-bb82-4931-bea0-67c67eea76fa" containerID="a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038" exitCode=0 Jul 28 00:04:34.359000 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.358840 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" Jul 28 00:04:34.359000 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.358864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" event={"ID":"a317b73f-bb82-4931-bea0-67c67eea76fa","Type":"ContainerDied","Data":"a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038"} Jul 28 00:04:34.359000 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.358917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" event={"ID":"a317b73f-bb82-4931-bea0-67c67eea76fa","Type":"ContainerDied","Data":"c677dfeb20658491188a0850c3e0fdc3dff21480bbb9fbb3aa59647d72640262"} Jul 28 00:04:34.359000 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.358939 2564 scope.go:117] "RemoveContainer" containerID="a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038" Jul 28 00:04:34.371338 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.371315 2564 scope.go:117] "RemoveContainer" containerID="a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038" Jul 28 00:04:34.371665 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:04:34.371636 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038\": container with ID starting with a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038 not found: ID does not exist" containerID="a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038" Jul 28 00:04:34.371737 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.371677 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038"} err="failed to get container status \"a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038\": rpc error: code = NotFound desc = could not find container \"a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038\": container with ID starting with a041067726c05fd66d392a08236d83d342497c6c015e629992e9454bfe220038 not found: ID does not exist" Jul 28 00:04:34.384378 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.384353 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx"] Jul 28 00:04:34.387631 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:34.387609 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx"] Jul 28 00:04:35.782079 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:35.782048 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a317b73f-bb82-4931-bea0-67c67eea76fa" path="/var/lib/kubelet/pods/a317b73f-bb82-4931-bea0-67c67eea76fa/volumes" Jul 28 00:04:38.873706 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:38.873665 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-4a8122/auth-test-model-4a8122-kserve-5466c574b7-zwzzx" podUID="a317b73f-bb82-4931-bea0-67c67eea76fa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 00:04:41.746893 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:41.746856 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:04:52.643925 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.643896 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5"] Jul 28 00:04:52.644357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.644214 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a317b73f-bb82-4931-bea0-67c67eea76fa" containerName="main" Jul 28 00:04:52.644357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.644224 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a317b73f-bb82-4931-bea0-67c67eea76fa" containerName="main" Jul 28 00:04:52.644357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.644237 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bcd7aa-9300-441b-b872-f5f5013abd0f" containerName="image-pruner" Jul 28 00:04:52.644357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.644243 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bcd7aa-9300-441b-b872-f5f5013abd0f" containerName="image-pruner" Jul 28 00:04:52.644357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.644313 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="24bcd7aa-9300-441b-b872-f5f5013abd0f" containerName="image-pruner" Jul 28 00:04:52.644357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.644320 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a317b73f-bb82-4931-bea0-67c67eea76fa" containerName="main" Jul 28 00:04:52.647259 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.647243 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.650354 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.650330 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"openshift-service-ca.crt\"" Jul 28 00:04:52.650354 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.650341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"default-dockercfg-gdmj9\"" Jul 28 00:04:52.650531 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.650341 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"sub-test-model-4a8122-kserve-self-signed-certs\"" Jul 28 00:04:52.650531 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.650396 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"kube-root-ca.crt\"" Jul 28 00:04:52.657700 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.657678 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5"] Jul 28 00:04:52.671369 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.671350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:04:52.759342 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.759318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-dshm\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.759451 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.759358 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-model-cache\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.759451 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.759381 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tls-certs\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.759451 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.759427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-home\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.759584 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.759463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tmp-dir\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.759584 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.759541 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjws\" (UniqueName: \"kubernetes.io/projected/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-kube-api-access-csjws\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860453 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tls-certs\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-home\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860483 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tmp-dir\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjws\" (UniqueName: \"kubernetes.io/projected/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-kube-api-access-csjws\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-dshm\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-model-cache\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.860915 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-home\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.861031 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tmp-dir\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.861031 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.860957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-model-cache\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.863068 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.863044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-dshm\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.863267 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.863253 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tls-certs\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.868286 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.868260 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjws\" (UniqueName: \"kubernetes.io/projected/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-kube-api-access-csjws\") pod \"sub-test-model-4a8122-kserve-5f656ddb47-ppwq5\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:52.956288 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:52.956227 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:04:53.085578 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:53.085555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5"] Jul 28 00:04:53.087025 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:04:53.086996 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4aaa887_f3ac_4d42_924e_e8bf71a2b8a9.slice/crio-f77851e53e73270d53774a13a418a45d2e3cead31395a7e3753415ec333572c3 WatchSource:0}: Error finding container f77851e53e73270d53774a13a418a45d2e3cead31395a7e3753415ec333572c3: Status 404 returned error can't find the container with id f77851e53e73270d53774a13a418a45d2e3cead31395a7e3753415ec333572c3 Jul 28 00:04:53.458392 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:53.458353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" event={"ID":"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9","Type":"ContainerStarted","Data":"3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac"} Jul 28 00:04:53.458392 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:53.458393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" event={"ID":"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9","Type":"ContainerStarted","Data":"f77851e53e73270d53774a13a418a45d2e3cead31395a7e3753415ec333572c3"} Jul 28 00:04:53.475164 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:53.475114 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" podStartSLOduration=1.475099625 podStartE2EDuration="1.475099625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:04:53.47288437 +0000 UTC m=+2936.270848996" watchObservedRunningTime="2026-07-28 00:04:53.475099625 +0000 UTC m=+2936.273064246" Jul 28 00:04:56.406294 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:04:56.406256 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:05:02.957205 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:05:02.957165 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:05:02.957687 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:05:02.957230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:05:02.969746 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:05:02.969721 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:05:03.521079 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:05:03.521045 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:05:27.438428 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:05:27.438392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:05:47.438628 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:05:47.438587 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:06:08.647781 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:08.647739 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:06:14.480072 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:14.480033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:15.633347 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:15.633318 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5"] Jul 28 00:06:16.753349 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:16.753321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:26.247318 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:26.247280 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:06:30.652703 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.652661 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" podUID="a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" containerName="main" containerID="cri-o://3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac" gracePeriod=45 Jul 28 00:06:30.904271 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.904215 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:06:30.981024 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.980990 2564 generic.go:356] "Generic (PLEG): container finished" podID="a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" containerID="3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac" exitCode=0 Jul 28 00:06:30.981173 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.981033 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" Jul 28 00:06:30.981173 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.981078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" event={"ID":"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9","Type":"ContainerDied","Data":"3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac"} Jul 28 00:06:30.981173 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.981121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5" event={"ID":"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9","Type":"ContainerDied","Data":"f77851e53e73270d53774a13a418a45d2e3cead31395a7e3753415ec333572c3"} Jul 28 00:06:30.981173 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.981143 2564 scope.go:117] "RemoveContainer" containerID="3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac" Jul 28 00:06:30.989842 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.989810 2564 scope.go:117] "RemoveContainer" containerID="3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac" Jul 28 00:06:30.990095 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:06:30.990075 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac\": container with ID starting with 3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac not found: ID does not exist" containerID="3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac" Jul 28 00:06:30.990158 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:30.990105 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac"} err="failed to get container status \"3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac\": rpc error: code = NotFound desc = could not find container \"3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac\": container with ID starting with 3b02b1d0d4e603bef5ec88d059b9e46fec4758971ae1515ff0f52a711c38deac not found: ID does not exist" Jul 28 00:06:31.012942 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.012926 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tls-certs\") pod \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " Jul 28 00:06:31.013008 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.012967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tmp-dir\") pod \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " Jul 28 00:06:31.013042 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013017 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-dshm\") pod \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " Jul 28 00:06:31.013075 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013045 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-home\") pod \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " Jul 28 00:06:31.013112 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013093 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-model-cache\") pod \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " Jul 28 00:06:31.013150 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013125 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csjws\" (UniqueName: \"kubernetes.io/projected/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-kube-api-access-csjws\") pod \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\" (UID: \"a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9\") " Jul 28 00:06:31.013325 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" (UID: "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:06:31.013441 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013382 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" (UID: "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:06:31.013441 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013393 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-home" (OuterVolumeSpecName: "home") pod "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" (UID: "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:06:31.013571 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013478 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tmp-dir\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:06:31.013571 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013496 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-home\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:06:31.013571 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.013509 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-model-cache\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:06:31.015208 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.015185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-dshm" (OuterVolumeSpecName: "dshm") pod "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" (UID: "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:06:31.015298 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.015216 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" (UID: "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:06:31.015298 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.015281 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-kube-api-access-csjws" (OuterVolumeSpecName: "kube-api-access-csjws") pod "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" (UID: "a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9"). InnerVolumeSpecName "kube-api-access-csjws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:06:31.114683 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.114657 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csjws\" (UniqueName: \"kubernetes.io/projected/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-kube-api-access-csjws\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:06:31.114683 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.114681 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-tls-certs\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:06:31.114864 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.114690 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9-dshm\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:06:31.304712 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.304684 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5"] Jul 28 00:06:31.308220 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.308186 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/sub-test-model-4a8122-kserve-5f656ddb47-ppwq5"] Jul 28 00:06:31.782941 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:31.782911 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" path="/var/lib/kubelet/pods/a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9/volumes" Jul 28 00:06:33.058594 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:33.058568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:36.542288 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:36.542253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:06:49.945198 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:49.945122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:06:49.991195 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:06:49.991163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:01.747497 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.747464 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh"] Jul 28 00:07:01.747903 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.747866 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" containerName="main" Jul 28 00:07:01.747903 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.747879 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" containerName="main" Jul 28 00:07:01.747984 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.747951 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4aaa887-f3ac-4d42-924e-e8bf71a2b8a9" containerName="main" Jul 28 00:07:01.752371 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.752350 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.755565 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.755543 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"rate-test-model-4a8122-kserve-self-signed-certs\"" Jul 28 00:07:01.755699 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.755565 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"kube-root-ca.crt\"" Jul 28 00:07:01.755699 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.755544 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"openshift-service-ca.crt\"" Jul 28 00:07:01.755699 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.755544 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4a8122\"/\"default-dockercfg-gdmj9\"" Jul 28 00:07:01.760863 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.760838 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh"] Jul 28 00:07:01.772721 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.772692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:07:01.856648 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.856621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-dshm\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.856648 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.856646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tls-certs\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.856793 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.856686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tmp-dir\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.856865 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.856857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-home\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.856908 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.856874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4wg\" (UniqueName: \"kubernetes.io/projected/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-kube-api-access-dn4wg\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.856908 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.856903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-model-cache\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958002 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.957978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-dshm\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958096 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tls-certs\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958096 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tmp-dir\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958096 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-home\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958252 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4wg\" (UniqueName: \"kubernetes.io/projected/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-kube-api-access-dn4wg\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958306 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-model-cache\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958449 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tmp-dir\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958510 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-home\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.958603 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.958583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-model-cache\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.960292 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.960268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-dshm\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.960551 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.960534 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tls-certs\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:01.966380 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:01.966363 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4wg\" (UniqueName: \"kubernetes.io/projected/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-kube-api-access-dn4wg\") pod \"rate-test-model-4a8122-kserve-67dd4994ff-w5gsh\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:02.062647 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:02.062620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:02.192837 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:02.192800 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh"] Jul 28 00:07:02.194743 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:07:02.194717 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3699a8e_f21d_4fd5_98e4_9759fe729ae5.slice/crio-94ace0c27478157c976a7c84da4f66789dd02a833275d644b62b1fa1790fbb51 WatchSource:0}: Error finding container 94ace0c27478157c976a7c84da4f66789dd02a833275d644b62b1fa1790fbb51: Status 404 returned error can't find the container with id 94ace0c27478157c976a7c84da4f66789dd02a833275d644b62b1fa1790fbb51 Jul 28 00:07:03.152218 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:03.152181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" event={"ID":"c3699a8e-f21d-4fd5-98e4-9759fe729ae5","Type":"ContainerStarted","Data":"0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066"} Jul 28 00:07:03.152736 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:03.152230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" event={"ID":"c3699a8e-f21d-4fd5-98e4-9759fe729ae5","Type":"ContainerStarted","Data":"94ace0c27478157c976a7c84da4f66789dd02a833275d644b62b1fa1790fbb51"} Jul 28 00:07:03.167596 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:03.167544 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" podStartSLOduration=2.167526151 podStartE2EDuration="2.167526151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:07:03.16698226 +0000 UTC m=+3065.964946897" watchObservedRunningTime="2026-07-28 00:07:03.167526151 +0000 UTC m=+3065.965490774" Jul 28 00:07:05.836211 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:05.836169 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:07:12.062826 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:12.062774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:12.065282 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:12.062854 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:12.075087 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:12.075068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:12.207158 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:12.207136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:07:34.644233 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:34.644196 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:07:54.539352 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:07:54.539316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:08:01.349200 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:01.349168 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:02.229291 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:02.229256 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh"] Jul 28 00:08:02.248055 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:02.248025 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:03.123615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.123584 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-139736274771120"] Jul 28 00:08:03.127140 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.127124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:03.129513 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.129487 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4k2qn\"" Jul 28 00:08:03.132304 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.132271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274771120"] Jul 28 00:08:03.151602 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.151578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqc8f\" (UniqueName: \"kubernetes.io/projected/4ae076bd-df4b-4db7-98bf-672a7aba6d4b-kube-api-access-dqc8f\") pod \"test-curl-7438-139736274771120\" (UID: \"4ae076bd-df4b-4db7-98bf-672a7aba6d4b\") " pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:03.252430 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.252399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqc8f\" (UniqueName: \"kubernetes.io/projected/4ae076bd-df4b-4db7-98bf-672a7aba6d4b-kube-api-access-dqc8f\") pod \"test-curl-7438-139736274771120\" (UID: \"4ae076bd-df4b-4db7-98bf-672a7aba6d4b\") " pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:03.260290 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.260271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqc8f\" (UniqueName: \"kubernetes.io/projected/4ae076bd-df4b-4db7-98bf-672a7aba6d4b-kube-api-access-dqc8f\") pod \"test-curl-7438-139736274771120\" (UID: \"4ae076bd-df4b-4db7-98bf-672a7aba6d4b\") " pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:03.438700 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.438630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:03.576172 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.576146 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274771120"] Jul 28 00:08:03.577606 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:03.577576 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ae076bd_df4b_4db7_98bf_672a7aba6d4b.slice/crio-15951ea3ff20df988c0bf722afb32353b1ad70fcecc3fed674f392a4d77a16d9 WatchSource:0}: Error finding container 15951ea3ff20df988c0bf722afb32353b1ad70fcecc3fed674f392a4d77a16d9: Status 404 returned error can't find the container with id 15951ea3ff20df988c0bf722afb32353b1ad70fcecc3fed674f392a4d77a16d9 Jul 28 00:08:03.579283 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:03.579263 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:08:04.483480 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:04.483444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274771120" event={"ID":"4ae076bd-df4b-4db7-98bf-672a7aba6d4b","Type":"ContainerStarted","Data":"15951ea3ff20df988c0bf722afb32353b1ad70fcecc3fed674f392a4d77a16d9"} Jul 28 00:08:05.489927 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:05.489897 2564 generic.go:356] "Generic (PLEG): container finished" podID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" containerID="ac82becfdbe4d4412141d21af62249bdccdc8f5669b533f1199ead893b8c2ee2" exitCode=0 Jul 28 00:08:05.490312 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:05.489982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274771120" event={"ID":"4ae076bd-df4b-4db7-98bf-672a7aba6d4b","Type":"ContainerDied","Data":"ac82becfdbe4d4412141d21af62249bdccdc8f5669b533f1199ead893b8c2ee2"} Jul 28 00:08:06.629351 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.629327 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:06.663115 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.663090 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7438-139736274771120_4ae076bd-df4b-4db7-98bf-672a7aba6d4b/test-curl-7438-139736274771120/0.log" Jul 28 00:08:06.682129 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.682102 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqc8f\" (UniqueName: \"kubernetes.io/projected/4ae076bd-df4b-4db7-98bf-672a7aba6d4b-kube-api-access-dqc8f\") pod \"4ae076bd-df4b-4db7-98bf-672a7aba6d4b\" (UID: \"4ae076bd-df4b-4db7-98bf-672a7aba6d4b\") " Jul 28 00:08:06.684269 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.684247 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae076bd-df4b-4db7-98bf-672a7aba6d4b-kube-api-access-dqc8f" (OuterVolumeSpecName: "kube-api-access-dqc8f") pod "4ae076bd-df4b-4db7-98bf-672a7aba6d4b" (UID: "4ae076bd-df4b-4db7-98bf-672a7aba6d4b"). InnerVolumeSpecName "kube-api-access-dqc8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:06.703795 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.703767 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-139736274771120"] Jul 28 00:08:06.731655 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.731634 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-139736274771120"] Jul 28 00:08:06.782916 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.782896 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqc8f\" (UniqueName: \"kubernetes.io/projected/4ae076bd-df4b-4db7-98bf-672a7aba6d4b-kube-api-access-dqc8f\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:06.876203 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.876177 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-139736274768944"] Jul 28 00:08:06.876544 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.876531 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" containerName="test-curl-7438-139736274771120" Jul 28 00:08:06.876585 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.876547 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" containerName="test-curl-7438-139736274771120" Jul 28 00:08:06.876629 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.876618 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" containerName="test-curl-7438-139736274771120" Jul 28 00:08:06.879544 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.879530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:06.912394 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.912371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274768944"] Jul 28 00:08:06.984582 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:06.984558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scl9g\" (UniqueName: \"kubernetes.io/projected/89c99a1b-5385-4480-a0ab-d3396c92b021-kube-api-access-scl9g\") pod \"test-curl-7438-139736274768944\" (UID: \"89c99a1b-5385-4480-a0ab-d3396c92b021\") " pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:07.085295 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.085205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scl9g\" (UniqueName: \"kubernetes.io/projected/89c99a1b-5385-4480-a0ab-d3396c92b021-kube-api-access-scl9g\") pod \"test-curl-7438-139736274768944\" (UID: \"89c99a1b-5385-4480-a0ab-d3396c92b021\") " pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:07.093185 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.093163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scl9g\" (UniqueName: \"kubernetes.io/projected/89c99a1b-5385-4480-a0ab-d3396c92b021-kube-api-access-scl9g\") pod \"test-curl-7438-139736274768944\" (UID: \"89c99a1b-5385-4480-a0ab-d3396c92b021\") " pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:07.188556 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.188516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:07.322045 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.322013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274768944"] Jul 28 00:08:07.323648 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:07.323623 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89c99a1b_5385_4480_a0ab_d3396c92b021.slice/crio-b3dd14350d1fdea6836161bc188081f3dff0857337b12d7d96dd148a362f34d8 WatchSource:0}: Error finding container b3dd14350d1fdea6836161bc188081f3dff0857337b12d7d96dd148a362f34d8: Status 404 returned error can't find the container with id b3dd14350d1fdea6836161bc188081f3dff0857337b12d7d96dd148a362f34d8 Jul 28 00:08:07.501421 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.501391 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15951ea3ff20df988c0bf722afb32353b1ad70fcecc3fed674f392a4d77a16d9" Jul 28 00:08:07.501421 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.501408 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:07.502619 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.502594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274768944" event={"ID":"89c99a1b-5385-4480-a0ab-d3396c92b021","Type":"ContainerStarted","Data":"b3dd14350d1fdea6836161bc188081f3dff0857337b12d7d96dd148a362f34d8"} Jul 28 00:08:07.512078 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:08:07.512054 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7438-139736274771120\" is forbidden: User \"system:node:ip-10-0-141-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-248.ec2.internal' and this object" podUID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:07.781357 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:08:07.781267 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7438-139736274771120\" is forbidden: User \"system:node:ip-10-0-141-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-248.ec2.internal' and this object" podUID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" pod="opendatahub/test-curl-7438-139736274771120" Jul 28 00:08:07.783033 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:07.783009 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae076bd-df4b-4db7-98bf-672a7aba6d4b" path="/var/lib/kubelet/pods/4ae076bd-df4b-4db7-98bf-672a7aba6d4b/volumes" Jul 28 00:08:08.508986 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:08.508953 2564 generic.go:356] "Generic (PLEG): container finished" podID="89c99a1b-5385-4480-a0ab-d3396c92b021" containerID="20b11b24eae1ab867e98607d8914a6fd57c0c4cdd7cba42bb0625af358dd9413" exitCode=0 Jul 28 00:08:08.509161 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:08.509048 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274768944" event={"ID":"89c99a1b-5385-4480-a0ab-d3396c92b021","Type":"ContainerDied","Data":"20b11b24eae1ab867e98607d8914a6fd57c0c4cdd7cba42bb0625af358dd9413"} Jul 28 00:08:09.655408 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.655380 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:09.672078 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.672035 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7438-139736274768944_89c99a1b-5385-4480-a0ab-d3396c92b021/test-curl-7438-139736274768944/0.log" Jul 28 00:08:09.695193 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.695168 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-139736274768944"] Jul 28 00:08:09.697278 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.697253 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-139736274768944"] Jul 28 00:08:09.706773 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.706754 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scl9g\" (UniqueName: \"kubernetes.io/projected/89c99a1b-5385-4480-a0ab-d3396c92b021-kube-api-access-scl9g\") pod \"89c99a1b-5385-4480-a0ab-d3396c92b021\" (UID: \"89c99a1b-5385-4480-a0ab-d3396c92b021\") " Jul 28 00:08:09.708930 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.708909 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c99a1b-5385-4480-a0ab-d3396c92b021-kube-api-access-scl9g" (OuterVolumeSpecName: "kube-api-access-scl9g") pod "89c99a1b-5385-4480-a0ab-d3396c92b021" (UID: "89c99a1b-5385-4480-a0ab-d3396c92b021"). InnerVolumeSpecName "kube-api-access-scl9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:09.782009 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.781950 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c99a1b-5385-4480-a0ab-d3396c92b021" path="/var/lib/kubelet/pods/89c99a1b-5385-4480-a0ab-d3396c92b021/volumes" Jul 28 00:08:09.807450 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.807419 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scl9g\" (UniqueName: \"kubernetes.io/projected/89c99a1b-5385-4480-a0ab-d3396c92b021-kube-api-access-scl9g\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:09.850773 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.850742 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:09.851164 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.851146 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c99a1b-5385-4480-a0ab-d3396c92b021" containerName="test-curl-7438-139736274768944" Jul 28 00:08:09.851242 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.851167 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c99a1b-5385-4480-a0ab-d3396c92b021" containerName="test-curl-7438-139736274768944" Jul 28 00:08:09.851303 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.851253 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c99a1b-5385-4480-a0ab-d3396c92b021" containerName="test-curl-7438-139736274768944" Jul 28 00:08:09.854165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.854147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:09.858036 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.858015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:09.907933 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:09.907905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpz9\" (UniqueName: \"kubernetes.io/projected/8be8919b-5d02-4123-a0c6-23b0049caad6-kube-api-access-zwpz9\") pod \"test-curl-7438-139736274786320\" (UID: \"8be8919b-5d02-4123-a0c6-23b0049caad6\") " pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:10.008682 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.008643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpz9\" (UniqueName: \"kubernetes.io/projected/8be8919b-5d02-4123-a0c6-23b0049caad6-kube-api-access-zwpz9\") pod \"test-curl-7438-139736274786320\" (UID: \"8be8919b-5d02-4123-a0c6-23b0049caad6\") " pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:10.016658 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.016630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpz9\" (UniqueName: \"kubernetes.io/projected/8be8919b-5d02-4123-a0c6-23b0049caad6-kube-api-access-zwpz9\") pod \"test-curl-7438-139736274786320\" (UID: \"8be8919b-5d02-4123-a0c6-23b0049caad6\") " pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:10.164304 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.164231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:10.290963 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.290930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:10.292580 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:10.292556 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8be8919b_5d02_4123_a0c6_23b0049caad6.slice/crio-5bd73dabb977aa1557e28380d69ce40abaf0bd85c585a86ff0e75788ae403792 WatchSource:0}: Error finding container 5bd73dabb977aa1557e28380d69ce40abaf0bd85c585a86ff0e75788ae403792: Status 404 returned error can't find the container with id 5bd73dabb977aa1557e28380d69ce40abaf0bd85c585a86ff0e75788ae403792 Jul 28 00:08:10.521142 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.521120 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274768944" Jul 28 00:08:10.521328 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.521143 2564 scope.go:117] "RemoveContainer" containerID="20b11b24eae1ab867e98607d8914a6fd57c0c4cdd7cba42bb0625af358dd9413" Jul 28 00:08:10.522626 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:10.522452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274786320" event={"ID":"8be8919b-5d02-4123-a0c6-23b0049caad6","Type":"ContainerStarted","Data":"5bd73dabb977aa1557e28380d69ce40abaf0bd85c585a86ff0e75788ae403792"} Jul 28 00:08:11.529949 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:11.529916 2564 generic.go:356] "Generic (PLEG): container finished" podID="8be8919b-5d02-4123-a0c6-23b0049caad6" containerID="ede620e3422a2983cc7c6a02d0c910b3b32e4f6027397d2781c40293e4ec3d32" exitCode=0 Jul 28 00:08:11.530331 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:11.529979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274786320" event={"ID":"8be8919b-5d02-4123-a0c6-23b0049caad6","Type":"ContainerDied","Data":"ede620e3422a2983cc7c6a02d0c910b3b32e4f6027397d2781c40293e4ec3d32"} Jul 28 00:08:12.141438 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.141401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:08:12.672596 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.672574 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:12.689637 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.689617 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7438-139736274786320_8be8919b-5d02-4123-a0c6-23b0049caad6/test-curl-7438-139736274786320/0.log" Jul 28 00:08:12.713838 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.712557 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:12.727569 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.727485 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:12.734142 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.734117 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwpz9\" (UniqueName: \"kubernetes.io/projected/8be8919b-5d02-4123-a0c6-23b0049caad6-kube-api-access-zwpz9\") pod \"8be8919b-5d02-4123-a0c6-23b0049caad6\" (UID: \"8be8919b-5d02-4123-a0c6-23b0049caad6\") " Jul 28 00:08:12.737310 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.737288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be8919b-5d02-4123-a0c6-23b0049caad6-kube-api-access-zwpz9" (OuterVolumeSpecName: "kube-api-access-zwpz9") pod "8be8919b-5d02-4123-a0c6-23b0049caad6" (UID: "8be8919b-5d02-4123-a0c6-23b0049caad6"). InnerVolumeSpecName "kube-api-access-zwpz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:12.819634 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.819565 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:12.819935 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.819921 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be8919b-5d02-4123-a0c6-23b0049caad6" containerName="test-curl-7438-139736274786320" Jul 28 00:08:12.819979 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.819936 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be8919b-5d02-4123-a0c6-23b0049caad6" containerName="test-curl-7438-139736274786320" Jul 28 00:08:12.820018 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.820009 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be8919b-5d02-4123-a0c6-23b0049caad6" containerName="test-curl-7438-139736274786320" Jul 28 00:08:12.822923 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.822908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:12.826848 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.826806 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:12.835869 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.835850 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpz9\" (UniqueName: \"kubernetes.io/projected/8be8919b-5d02-4123-a0c6-23b0049caad6-kube-api-access-zwpz9\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:12.936388 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:12.936359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qftp\" (UniqueName: \"kubernetes.io/projected/79d68cb4-e457-4860-92d3-b2b587db9a44-kube-api-access-2qftp\") pod \"test-curl-7438-139736274786320\" (UID: \"79d68cb4-e457-4860-92d3-b2b587db9a44\") " pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:13.037328 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.037280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qftp\" (UniqueName: \"kubernetes.io/projected/79d68cb4-e457-4860-92d3-b2b587db9a44-kube-api-access-2qftp\") pod \"test-curl-7438-139736274786320\" (UID: \"79d68cb4-e457-4860-92d3-b2b587db9a44\") " pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:13.045537 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.045504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qftp\" (UniqueName: \"kubernetes.io/projected/79d68cb4-e457-4860-92d3-b2b587db9a44-kube-api-access-2qftp\") pod \"test-curl-7438-139736274786320\" (UID: \"79d68cb4-e457-4860-92d3-b2b587db9a44\") " pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:13.134409 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.134308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:13.266357 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.266333 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:13.268155 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:13.268131 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79d68cb4_e457_4860_92d3_b2b587db9a44.slice/crio-56d244bd72898933ed0f0236244b430f6c94b04dc9afebaeae84746d9879b228 WatchSource:0}: Error finding container 56d244bd72898933ed0f0236244b430f6c94b04dc9afebaeae84746d9879b228: Status 404 returned error can't find the container with id 56d244bd72898933ed0f0236244b430f6c94b04dc9afebaeae84746d9879b228 Jul 28 00:08:13.542554 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.542516 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:13.542754 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.542525 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bd73dabb977aa1557e28380d69ce40abaf0bd85c585a86ff0e75788ae403792" Jul 28 00:08:13.543752 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.543717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274786320" event={"ID":"79d68cb4-e457-4860-92d3-b2b587db9a44","Type":"ContainerStarted","Data":"56d244bd72898933ed0f0236244b430f6c94b04dc9afebaeae84746d9879b228"} Jul 28 00:08:13.553896 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.553867 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7438-139736274786320" oldPodUID="8be8919b-5d02-4123-a0c6-23b0049caad6" podUID="79d68cb4-e457-4860-92d3-b2b587db9a44" Jul 28 00:08:13.783108 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:13.783029 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be8919b-5d02-4123-a0c6-23b0049caad6" path="/var/lib/kubelet/pods/8be8919b-5d02-4123-a0c6-23b0049caad6/volumes" Jul 28 00:08:14.550347 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:14.550315 2564 generic.go:356] "Generic (PLEG): container finished" podID="79d68cb4-e457-4860-92d3-b2b587db9a44" containerID="bc61a0346ba0c95f7f9ecc586ed15b0430fba226452055b244273c810363d579" exitCode=0 Jul 28 00:08:14.550522 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:14.550388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274786320" event={"ID":"79d68cb4-e457-4860-92d3-b2b587db9a44","Type":"ContainerDied","Data":"bc61a0346ba0c95f7f9ecc586ed15b0430fba226452055b244273c810363d579"} Jul 28 00:08:15.690235 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.690212 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:15.708106 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.708076 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7438-139736274786320_79d68cb4-e457-4860-92d3-b2b587db9a44/test-curl-7438-139736274786320/0.log" Jul 28 00:08:15.729976 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.729945 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:15.732087 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.732067 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-139736274786320"] Jul 28 00:08:15.761835 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.761788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qftp\" (UniqueName: \"kubernetes.io/projected/79d68cb4-e457-4860-92d3-b2b587db9a44-kube-api-access-2qftp\") pod \"79d68cb4-e457-4860-92d3-b2b587db9a44\" (UID: \"79d68cb4-e457-4860-92d3-b2b587db9a44\") " Jul 28 00:08:15.764047 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.764024 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d68cb4-e457-4860-92d3-b2b587db9a44-kube-api-access-2qftp" (OuterVolumeSpecName: "kube-api-access-2qftp") pod "79d68cb4-e457-4860-92d3-b2b587db9a44" (UID: "79d68cb4-e457-4860-92d3-b2b587db9a44"). InnerVolumeSpecName "kube-api-access-2qftp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:15.782869 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.782842 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79d68cb4-e457-4860-92d3-b2b587db9a44" path="/var/lib/kubelet/pods/79d68cb4-e457-4860-92d3-b2b587db9a44/volumes" Jul 28 00:08:15.849811 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.849737 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:15.850155 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.850137 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79d68cb4-e457-4860-92d3-b2b587db9a44" containerName="test-curl-7438-139736274786320" Jul 28 00:08:15.850230 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.850158 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="79d68cb4-e457-4860-92d3-b2b587db9a44" containerName="test-curl-7438-139736274786320" Jul 28 00:08:15.850291 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.850281 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="79d68cb4-e457-4860-92d3-b2b587db9a44" containerName="test-curl-7438-139736274786320" Jul 28 00:08:15.853374 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.853355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:15.859099 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.859077 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:15.862872 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.862850 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qftp\" (UniqueName: \"kubernetes.io/projected/79d68cb4-e457-4860-92d3-b2b587db9a44-kube-api-access-2qftp\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:15.963773 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:15.963726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwbh\" (UniqueName: \"kubernetes.io/projected/54262128-c9b6-4221-9b35-6d2e04fe8be6-kube-api-access-fmwbh\") pod \"test-curl-7438-139736274787328\" (UID: \"54262128-c9b6-4221-9b35-6d2e04fe8be6\") " pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:16.064220 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.064189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwbh\" (UniqueName: \"kubernetes.io/projected/54262128-c9b6-4221-9b35-6d2e04fe8be6-kube-api-access-fmwbh\") pod \"test-curl-7438-139736274787328\" (UID: \"54262128-c9b6-4221-9b35-6d2e04fe8be6\") " pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:16.071700 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.071668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwbh\" (UniqueName: \"kubernetes.io/projected/54262128-c9b6-4221-9b35-6d2e04fe8be6-kube-api-access-fmwbh\") pod \"test-curl-7438-139736274787328\" (UID: \"54262128-c9b6-4221-9b35-6d2e04fe8be6\") " pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:16.163804 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.163683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:16.306001 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.305977 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:16.308504 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:16.308471 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54262128_c9b6_4221_9b35_6d2e04fe8be6.slice/crio-9b8852880650effff49a92da2822c983a5fbf756c0d1182bf5fcd77b03470b86 WatchSource:0}: Error finding container 9b8852880650effff49a92da2822c983a5fbf756c0d1182bf5fcd77b03470b86: Status 404 returned error can't find the container with id 9b8852880650effff49a92da2822c983a5fbf756c0d1182bf5fcd77b03470b86 Jul 28 00:08:16.564745 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.564707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274787328" event={"ID":"54262128-c9b6-4221-9b35-6d2e04fe8be6","Type":"ContainerStarted","Data":"9b8852880650effff49a92da2822c983a5fbf756c0d1182bf5fcd77b03470b86"} Jul 28 00:08:16.565911 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.565892 2564 scope.go:117] "RemoveContainer" containerID="bc61a0346ba0c95f7f9ecc586ed15b0430fba226452055b244273c810363d579" Jul 28 00:08:16.566043 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:16.565911 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274786320" Jul 28 00:08:17.248740 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.248704 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" podUID="c3699a8e-f21d-4fd5-98e4-9759fe729ae5" containerName="main" containerID="cri-o://0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066" gracePeriod=45 Jul 28 00:08:17.509619 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.509595 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:08:17.574052 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.574017 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3699a8e-f21d-4fd5-98e4-9759fe729ae5" containerID="0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066" exitCode=0 Jul 28 00:08:17.574225 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.574052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" event={"ID":"c3699a8e-f21d-4fd5-98e4-9759fe729ae5","Type":"ContainerDied","Data":"0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066"} Jul 28 00:08:17.574225 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.574098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" event={"ID":"c3699a8e-f21d-4fd5-98e4-9759fe729ae5","Type":"ContainerDied","Data":"94ace0c27478157c976a7c84da4f66789dd02a833275d644b62b1fa1790fbb51"} Jul 28 00:08:17.574225 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.574121 2564 scope.go:117] "RemoveContainer" containerID="0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066" Jul 28 00:08:17.574225 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.574064 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh" Jul 28 00:08:17.575083 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575062 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn4wg\" (UniqueName: \"kubernetes.io/projected/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-kube-api-access-dn4wg\") pod \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " Jul 28 00:08:17.575208 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575111 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tmp-dir\") pod \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " Jul 28 00:08:17.575208 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-dshm\") pod \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " Jul 28 00:08:17.575208 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575187 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-home\") pod \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " Jul 28 00:08:17.575707 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575215 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tls-certs\") pod \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " Jul 28 00:08:17.575707 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575300 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-model-cache\") pod \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\" (UID: \"c3699a8e-f21d-4fd5-98e4-9759fe729ae5\") " Jul 28 00:08:17.575707 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575424 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3699a8e-f21d-4fd5-98e4-9759fe729ae5" (UID: "c3699a8e-f21d-4fd5-98e4-9759fe729ae5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:08:17.575707 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575542 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-home" (OuterVolumeSpecName: "home") pod "c3699a8e-f21d-4fd5-98e4-9759fe729ae5" (UID: "c3699a8e-f21d-4fd5-98e4-9759fe729ae5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:08:17.576038 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575716 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3699a8e-f21d-4fd5-98e4-9759fe729ae5" (UID: "c3699a8e-f21d-4fd5-98e4-9759fe729ae5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:08:17.576038 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575840 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-model-cache\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:17.576038 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575857 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tmp-dir\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:17.576038 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.575869 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-home\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:17.576251 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.576122 2564 generic.go:356] "Generic (PLEG): container finished" podID="54262128-c9b6-4221-9b35-6d2e04fe8be6" containerID="e5d6f74b3424c9abb69e14b78705abdfd35c5c10c9994c790f6b07cfe56e979d" exitCode=0 Jul 28 00:08:17.576251 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.576229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274787328" event={"ID":"54262128-c9b6-4221-9b35-6d2e04fe8be6","Type":"ContainerDied","Data":"e5d6f74b3424c9abb69e14b78705abdfd35c5c10c9994c790f6b07cfe56e979d"} Jul 28 00:08:17.577789 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.577768 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-kube-api-access-dn4wg" (OuterVolumeSpecName: "kube-api-access-dn4wg") pod "c3699a8e-f21d-4fd5-98e4-9759fe729ae5" (UID: "c3699a8e-f21d-4fd5-98e4-9759fe729ae5"). InnerVolumeSpecName "kube-api-access-dn4wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:17.578183 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.578165 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-dshm" (OuterVolumeSpecName: "dshm") pod "c3699a8e-f21d-4fd5-98e4-9759fe729ae5" (UID: "c3699a8e-f21d-4fd5-98e4-9759fe729ae5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:08:17.578296 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.578278 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3699a8e-f21d-4fd5-98e4-9759fe729ae5" (UID: "c3699a8e-f21d-4fd5-98e4-9759fe729ae5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:08:17.583615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.583600 2564 scope.go:117] "RemoveContainer" containerID="0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066" Jul 28 00:08:17.583914 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:08:17.583897 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066\": container with ID starting with 0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066 not found: ID does not exist" containerID="0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066" Jul 28 00:08:17.583954 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.583923 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066"} err="failed to get container status \"0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066\": rpc error: code = NotFound desc = could not find container \"0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066\": container with ID starting with 0fe374054f7c6794cd9d323ee28f029f332e0a54df20caa3a56055d45d229066 not found: ID does not exist" Jul 28 00:08:17.677302 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.677276 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4wg\" (UniqueName: \"kubernetes.io/projected/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-kube-api-access-dn4wg\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:17.677302 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.677303 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-dshm\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:17.677450 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.677313 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3699a8e-f21d-4fd5-98e4-9759fe729ae5-tls-certs\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:17.888680 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.888637 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh"] Jul 28 00:08:17.892330 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:17.892303 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4a8122/rate-test-model-4a8122-kserve-67dd4994ff-w5gsh"] Jul 28 00:08:18.719310 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.719287 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:18.736585 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.736559 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7438-139736274787328_54262128-c9b6-4221-9b35-6d2e04fe8be6/test-curl-7438-139736274787328/0.log" Jul 28 00:08:18.758865 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.758841 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:18.760856 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.760837 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:18.787141 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.787118 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmwbh\" (UniqueName: \"kubernetes.io/projected/54262128-c9b6-4221-9b35-6d2e04fe8be6-kube-api-access-fmwbh\") pod \"54262128-c9b6-4221-9b35-6d2e04fe8be6\" (UID: \"54262128-c9b6-4221-9b35-6d2e04fe8be6\") " Jul 28 00:08:18.789261 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.789241 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54262128-c9b6-4221-9b35-6d2e04fe8be6-kube-api-access-fmwbh" (OuterVolumeSpecName: "kube-api-access-fmwbh") pod "54262128-c9b6-4221-9b35-6d2e04fe8be6" (UID: "54262128-c9b6-4221-9b35-6d2e04fe8be6"). InnerVolumeSpecName "kube-api-access-fmwbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:18.876456 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876388 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:18.876718 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876706 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54262128-c9b6-4221-9b35-6d2e04fe8be6" containerName="test-curl-7438-139736274787328" Jul 28 00:08:18.876757 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876720 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="54262128-c9b6-4221-9b35-6d2e04fe8be6" containerName="test-curl-7438-139736274787328" Jul 28 00:08:18.876757 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876739 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3699a8e-f21d-4fd5-98e4-9759fe729ae5" containerName="main" Jul 28 00:08:18.876757 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876744 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3699a8e-f21d-4fd5-98e4-9759fe729ae5" containerName="main" Jul 28 00:08:18.876881 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876799 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="54262128-c9b6-4221-9b35-6d2e04fe8be6" containerName="test-curl-7438-139736274787328" Jul 28 00:08:18.876881 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.876809 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3699a8e-f21d-4fd5-98e4-9759fe729ae5" containerName="main" Jul 28 00:08:18.879794 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.879779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:18.883515 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.883492 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:18.887912 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.887891 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmwbh\" (UniqueName: \"kubernetes.io/projected/54262128-c9b6-4221-9b35-6d2e04fe8be6-kube-api-access-fmwbh\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:18.988345 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:18.988309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv5ll\" (UniqueName: \"kubernetes.io/projected/aad0e2a0-0a39-429a-89e6-80163a64f891-kube-api-access-wv5ll\") pod \"test-curl-7438-139736274787328\" (UID: \"aad0e2a0-0a39-429a-89e6-80163a64f891\") " pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:19.089667 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.089632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv5ll\" (UniqueName: \"kubernetes.io/projected/aad0e2a0-0a39-429a-89e6-80163a64f891-kube-api-access-wv5ll\") pod \"test-curl-7438-139736274787328\" (UID: \"aad0e2a0-0a39-429a-89e6-80163a64f891\") " pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:19.097702 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.097680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv5ll\" (UniqueName: \"kubernetes.io/projected/aad0e2a0-0a39-429a-89e6-80163a64f891-kube-api-access-wv5ll\") pod \"test-curl-7438-139736274787328\" (UID: \"aad0e2a0-0a39-429a-89e6-80163a64f891\") " pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:19.190152 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.190070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:19.324376 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.324351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:19.326450 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:19.326420 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaad0e2a0_0a39_429a_89e6_80163a64f891.slice/crio-f30cd250d51196e26af4cb6098d3a1abd4dd3f5c0913bdfbdfa4eed5849b8819 WatchSource:0}: Error finding container f30cd250d51196e26af4cb6098d3a1abd4dd3f5c0913bdfbdfa4eed5849b8819: Status 404 returned error can't find the container with id f30cd250d51196e26af4cb6098d3a1abd4dd3f5c0913bdfbdfa4eed5849b8819 Jul 28 00:08:19.589955 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.589925 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:19.590126 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.589924 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8852880650effff49a92da2822c983a5fbf756c0d1182bf5fcd77b03470b86" Jul 28 00:08:19.591104 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.591075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274787328" event={"ID":"aad0e2a0-0a39-429a-89e6-80163a64f891","Type":"ContainerStarted","Data":"f30cd250d51196e26af4cb6098d3a1abd4dd3f5c0913bdfbdfa4eed5849b8819"} Jul 28 00:08:19.601488 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.601464 2564 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7438-139736274787328" oldPodUID="54262128-c9b6-4221-9b35-6d2e04fe8be6" podUID="aad0e2a0-0a39-429a-89e6-80163a64f891" Jul 28 00:08:19.782769 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.782736 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54262128-c9b6-4221-9b35-6d2e04fe8be6" path="/var/lib/kubelet/pods/54262128-c9b6-4221-9b35-6d2e04fe8be6/volumes" Jul 28 00:08:19.783185 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:19.783170 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3699a8e-f21d-4fd5-98e4-9759fe729ae5" path="/var/lib/kubelet/pods/c3699a8e-f21d-4fd5-98e4-9759fe729ae5/volumes" Jul 28 00:08:20.597367 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:20.597334 2564 generic.go:356] "Generic (PLEG): container finished" podID="aad0e2a0-0a39-429a-89e6-80163a64f891" containerID="ebdd0da5a67cf675a684fc827147a8a1c2b573579e7dd85b38c866c555f63cdc" exitCode=0 Jul 28 00:08:20.597547 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:20.597436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7438-139736274787328" event={"ID":"aad0e2a0-0a39-429a-89e6-80163a64f891","Type":"ContainerDied","Data":"ebdd0da5a67cf675a684fc827147a8a1c2b573579e7dd85b38c866c555f63cdc"} Jul 28 00:08:21.754937 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.754913 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:21.773112 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.773086 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7438-139736274787328_aad0e2a0-0a39-429a-89e6-80163a64f891/test-curl-7438-139736274787328/0.log" Jul 28 00:08:21.795999 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.795967 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:21.799168 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.799144 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7438-139736274787328"] Jul 28 00:08:21.814213 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.814191 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv5ll\" (UniqueName: \"kubernetes.io/projected/aad0e2a0-0a39-429a-89e6-80163a64f891-kube-api-access-wv5ll\") pod \"aad0e2a0-0a39-429a-89e6-80163a64f891\" (UID: \"aad0e2a0-0a39-429a-89e6-80163a64f891\") " Jul 28 00:08:21.816359 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.816339 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad0e2a0-0a39-429a-89e6-80163a64f891-kube-api-access-wv5ll" (OuterVolumeSpecName: "kube-api-access-wv5ll") pod "aad0e2a0-0a39-429a-89e6-80163a64f891" (UID: "aad0e2a0-0a39-429a-89e6-80163a64f891"). InnerVolumeSpecName "kube-api-access-wv5ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:08:21.915692 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:21.915608 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv5ll\" (UniqueName: \"kubernetes.io/projected/aad0e2a0-0a39-429a-89e6-80163a64f891-kube-api-access-wv5ll\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:08:22.273495 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:22.273461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:22.563884 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:22.563775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:22.616340 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:22.616308 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:22.616340 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:22.616341 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f30cd250d51196e26af4cb6098d3a1abd4dd3f5c0913bdfbdfa4eed5849b8819" Jul 28 00:08:22.626013 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:08:22.625980 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7438-139736274787328\" is forbidden: User \"system:node:ip-10-0-141-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-248.ec2.internal' and this object" podUID="aad0e2a0-0a39-429a-89e6-80163a64f891" pod="opendatahub/test-curl-7438-139736274787328" Jul 28 00:08:22.662423 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:22.662396 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:23.781784 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:23.781753 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aad0e2a0-0a39-429a-89e6-80163a64f891" path="/var/lib/kubelet/pods/aad0e2a0-0a39-429a-89e6-80163a64f891/volumes" Jul 28 00:08:25.725572 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:25.725534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:25.827627 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:25.827598 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:25.939560 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:25.939530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:26.129952 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.129917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:26.485866 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.485763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:26.581741 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.581706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:26.622780 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.622748 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg"] Jul 28 00:08:26.623121 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.623108 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aad0e2a0-0a39-429a-89e6-80163a64f891" containerName="test-curl-7438-139736274787328" Jul 28 00:08:26.623166 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.623123 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aad0e2a0-0a39-429a-89e6-80163a64f891" containerName="test-curl-7438-139736274787328" Jul 28 00:08:26.623200 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.623182 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aad0e2a0-0a39-429a-89e6-80163a64f891" containerName="test-curl-7438-139736274787328" Jul 28 00:08:26.626501 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.626482 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.628715 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.628695 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4a6d94-dockercfg-dgg4h\"" Jul 28 00:08:26.628806 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.628714 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4a6d94\"" Jul 28 00:08:26.634622 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.634601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg"] Jul 28 00:08:26.758782 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.758751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ctv4\" (UniqueName: \"kubernetes.io/projected/06096818-b72c-4535-b6fa-0c90474dcaa3-kube-api-access-8ctv4\") pod \"payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.759144 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.758794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06096818-b72c-4535-b6fa-0c90474dcaa3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.859688 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.859660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06096818-b72c-4535-b6fa-0c90474dcaa3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.859955 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.859928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ctv4\" (UniqueName: \"kubernetes.io/projected/06096818-b72c-4535-b6fa-0c90474dcaa3-kube-api-access-8ctv4\") pod \"payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.860418 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.860395 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06096818-b72c-4535-b6fa-0c90474dcaa3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.867540 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.867517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ctv4\" (UniqueName: \"kubernetes.io/projected/06096818-b72c-4535-b6fa-0c90474dcaa3-kube-api-access-8ctv4\") pod \"payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:26.938091 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:26.938067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:27.067033 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:27.067006 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg"] Jul 28 00:08:27.068470 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:27.068436 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06096818_b72c_4535_b6fa_0c90474dcaa3.slice/crio-c68692ff93d92e1b25750a417aafd2d2241c9e65c8e87c438b18ace75acf1a56 WatchSource:0}: Error finding container c68692ff93d92e1b25750a417aafd2d2241c9e65c8e87c438b18ace75acf1a56: Status 404 returned error can't find the container with id c68692ff93d92e1b25750a417aafd2d2241c9e65c8e87c438b18ace75acf1a56 Jul 28 00:08:27.645505 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:27.645472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" event={"ID":"06096818-b72c-4535-b6fa-0c90474dcaa3","Type":"ContainerStarted","Data":"c68692ff93d92e1b25750a417aafd2d2241c9e65c8e87c438b18ace75acf1a56"} Jul 28 00:08:29.657184 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:29.657148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" event={"ID":"06096818-b72c-4535-b6fa-0c90474dcaa3","Type":"ContainerStarted","Data":"4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e"} Jul 28 00:08:29.657591 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:29.657255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:29.672800 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:29.672744 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" podStartSLOduration=2.130747354 podStartE2EDuration="3.672728723s" totalImagesPullingTime="1.541981369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:29.672193606 +0000 UTC m=+3152.470158247" watchObservedRunningTime="2026-07-28 00:08:29.672728723 +0000 UTC m=+3152.470693348" Jul 28 00:08:30.212885 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:30.212851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:34.838785 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:34.838754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:08:40.664656 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:40.664625 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:08:46.209863 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:46.209805 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:46.426783 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:46.426746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:46.532150 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:46.532120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:46.631502 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:46.631465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:49.846044 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:49.846009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:49.946155 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:49.946121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:50.099062 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.098988 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd"] Jul 28 00:08:50.104516 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.104497 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.106768 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.106738 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7chpm\"" Jul 28 00:08:50.106768 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.106760 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 00:08:50.107495 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.107476 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-87c90a-serving-cert\"" Jul 28 00:08:50.107561 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.107522 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 00:08:50.111530 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.111507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd"] Jul 28 00:08:50.135918 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.135884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:50.254447 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.254411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:50.255565 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.255543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.255628 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.255582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.255628 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.255614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-kube-api-access-rnsgs\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.356406 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.356326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-kube-api-access-rnsgs\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.356524 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.356435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.356524 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.356475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.357103 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.357080 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.359082 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.359055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.364056 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.364034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-kube-api-access-rnsgs\") pod \"maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.416962 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.416942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:50.554672 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.554642 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd"] Jul 28 00:08:50.556072 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:50.556047 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc0d4c2_150f_4449_a238_6e6f831d1fc6.slice/crio-45e1855ccce72e7c7326e098a6c083d4ae854ce3fbd7d6e29d577185097ae9bc WatchSource:0}: Error finding container 45e1855ccce72e7c7326e098a6c083d4ae854ce3fbd7d6e29d577185097ae9bc: Status 404 returned error can't find the container with id 45e1855ccce72e7c7326e098a6c083d4ae854ce3fbd7d6e29d577185097ae9bc Jul 28 00:08:50.567159 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.567133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:50.668412 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.668348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:50.674369 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.674343 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb"] Jul 28 00:08:50.679017 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.678998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.681605 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.681571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-87c90a-dockercfg-fjfzw\"" Jul 28 00:08:50.681756 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.681740 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-87c90a\"" Jul 28 00:08:50.691149 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.691128 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb"] Jul 28 00:08:50.701229 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.701209 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc"] Jul 28 00:08:50.704575 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.704558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.715650 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.715626 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc"] Jul 28 00:08:50.760152 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.760131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-kube-api-access-bth24\") pod \"payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.760245 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.760164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.769872 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.769844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" event={"ID":"ecc0d4c2-150f-4449-a238-6e6f831d1fc6","Type":"ContainerStarted","Data":"45e1855ccce72e7c7326e098a6c083d4ae854ce3fbd7d6e29d577185097ae9bc"} Jul 28 00:08:50.861452 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.861429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.861876 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.861496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr77f\" (UniqueName: \"kubernetes.io/projected/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-kube-api-access-tr77f\") pod \"payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.861876 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.861597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.861876 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.861680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-kube-api-access-bth24\") pod \"payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.862259 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.862240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.869420 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.869395 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-kube-api-access-bth24\") pod \"payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:50.962457 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.962381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr77f\" (UniqueName: \"kubernetes.io/projected/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-kube-api-access-tr77f\") pod \"payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.962457 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.962429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.963292 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.963267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.970541 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.970509 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr77f\" (UniqueName: \"kubernetes.io/projected/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-kube-api-access-tr77f\") pod \"payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:50.991104 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:50.991082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:51.015354 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.015324 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:51.156940 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.156913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb"] Jul 28 00:08:51.160418 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:51.160390 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a20d1ba_180f_4884_85bf_c2148ee2bfe4.slice/crio-4f6e36e1037ddea0254ec592e6d30b8f58ee00c5ef746e3b5b6169026831bf8d WatchSource:0}: Error finding container 4f6e36e1037ddea0254ec592e6d30b8f58ee00c5ef746e3b5b6169026831bf8d: Status 404 returned error can't find the container with id 4f6e36e1037ddea0254ec592e6d30b8f58ee00c5ef746e3b5b6169026831bf8d Jul 28 00:08:51.186118 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.186082 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc"] Jul 28 00:08:51.188141 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:08:51.188116 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb5fbbf_8eb7_49c7_8772_43a7f0726c27.slice/crio-592cf31b1508dce6050e9419d7e2234253445f41c45b39676bf561346dbf1e1a WatchSource:0}: Error finding container 592cf31b1508dce6050e9419d7e2234253445f41c45b39676bf561346dbf1e1a: Status 404 returned error can't find the container with id 592cf31b1508dce6050e9419d7e2234253445f41c45b39676bf561346dbf1e1a Jul 28 00:08:51.784972 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.784945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:08:51.785165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.784997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" event={"ID":"2a20d1ba-180f-4884-85bf-c2148ee2bfe4","Type":"ContainerStarted","Data":"b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627"} Jul 28 00:08:51.785165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.785028 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:08:51.785165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.785050 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" event={"ID":"2a20d1ba-180f-4884-85bf-c2148ee2bfe4","Type":"ContainerStarted","Data":"4f6e36e1037ddea0254ec592e6d30b8f58ee00c5ef746e3b5b6169026831bf8d"} Jul 28 00:08:51.785165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.785069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" event={"ID":"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27","Type":"ContainerStarted","Data":"f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264"} Jul 28 00:08:51.785165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.785091 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" event={"ID":"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27","Type":"ContainerStarted","Data":"592cf31b1508dce6050e9419d7e2234253445f41c45b39676bf561346dbf1e1a"} Jul 28 00:08:51.794970 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.794920 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" podStartSLOduration=1.7949056890000001 podStartE2EDuration="1.794905689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:51.793122384 +0000 UTC m=+3174.591087019" watchObservedRunningTime="2026-07-28 00:08:51.794905689 +0000 UTC m=+3174.592870313" Jul 28 00:08:51.812972 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:51.812925 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" podStartSLOduration=1.8129067399999998 podStartE2EDuration="1.81290674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:51.811796812 +0000 UTC m=+3174.609761439" watchObservedRunningTime="2026-07-28 00:08:51.81290674 +0000 UTC m=+3174.610871392" Jul 28 00:08:53.342620 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:53.342581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:08:53.794005 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:53.793966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" event={"ID":"ecc0d4c2-150f-4449-a238-6e6f831d1fc6","Type":"ContainerStarted","Data":"86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1"} Jul 28 00:08:53.794193 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:53.794034 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:08:53.810745 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:53.810699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" podStartSLOduration=1.593573958 podStartE2EDuration="3.81068626s" totalImagesPullingTime="2.217112302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:08:53.80830048 +0000 UTC m=+3176.606265104" watchObservedRunningTime="2026-07-28 00:08:53.81068626 +0000 UTC m=+3176.608650884" Jul 28 00:08:54.322272 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:54.322218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:08:59.805927 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:08:59.805899 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:09:02.788660 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:02.788630 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:09:02.789121 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:02.788735 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:09:11.136523 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:11.136482 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:11.448176 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:11.448100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc_5fb5fbbf-8eb7-49c7-8772-43a7f0726c27/payload-processing/0.log" Jul 28 00:09:17.473765 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:17.473734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:09:21.644841 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:21.644781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:09:51.339233 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:51.339151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:09:55.189836 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:09:55.189788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:00.203924 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:00.203887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:03.284946 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:03.284908 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg_06096818-b72c-4535-b6fa-0c90474dcaa3/payload-processing/0.log" Jul 28 00:10:03.521746 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:03.521713 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc_5fb5fbbf-8eb7-49c7-8772-43a7f0726c27/payload-processing/0.log" Jul 28 00:10:03.785439 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:03.785412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:04.081005 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:04.080924 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:04.695609 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:04.695573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:06.771586 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:06.771551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:06.871337 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:06.871304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:06.999808 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:06.999776 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6"] Jul 28 00:10:07.004528 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.004503 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.006921 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.006892 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a35fc9-serving-cert\"" Jul 28 00:10:07.010890 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.010864 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6"] Jul 28 00:10:07.030851 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.030770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:07.102416 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.102388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.102537 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.102518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.102598 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.102563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgg2m\" (UniqueName: \"kubernetes.io/projected/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-kube-api-access-kgg2m\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.203858 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.203834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.203964 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.203869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgg2m\" (UniqueName: \"kubernetes.io/projected/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-kube-api-access-kgg2m\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.203964 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.203895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.204036 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:10:07.203991 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-serving-cert: secret "maas-api-e2e-ipp-cleanup-a35fc9-serving-cert" not found Jul 28 00:10:07.204075 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:10:07.204060 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls podName:6fa4a0f0-5a5e-46e1-825a-143ff8f4e219 nodeName:}" failed. No retries permitted until 2026-07-28 00:10:07.70404012 +0000 UTC m=+3250.502004723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" (UID: "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219") : secret "maas-api-e2e-ipp-cleanup-a35fc9-serving-cert" not found Jul 28 00:10:07.204455 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.204440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.211728 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.211710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgg2m\" (UniqueName: \"kubernetes.io/projected/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-kube-api-access-kgg2m\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.523250 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.523217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:07.622577 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.622550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:07.707305 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.707269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.707454 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.707381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:07.709767 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.709750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:07.918099 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:07.918007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:08.049456 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:08.049431 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6"] Jul 28 00:10:08.052304 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:10:08.052251 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa4a0f0_5a5e_46e1_825a_143ff8f4e219.slice/crio-7f142ba792ad9270d1020e4f550a8477c0dac8448d21f8b67849309198f01340 WatchSource:0}: Error finding container 7f142ba792ad9270d1020e4f550a8477c0dac8448d21f8b67849309198f01340: Status 404 returned error can't find the container with id 7f142ba792ad9270d1020e4f550a8477c0dac8448d21f8b67849309198f01340 Jul 28 00:10:08.206307 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:08.206227 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" event={"ID":"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219","Type":"ContainerStarted","Data":"7f142ba792ad9270d1020e4f550a8477c0dac8448d21f8b67849309198f01340"} Jul 28 00:10:09.212981 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:09.212947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" event={"ID":"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219","Type":"ContainerStarted","Data":"a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b"} Jul 28 00:10:09.213428 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:09.213001 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:09.229628 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:09.229582 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" podStartSLOduration=2.726214986 podStartE2EDuration="3.229565708s" totalImagesPullingTime="503.350722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:10:09.227628856 +0000 UTC m=+3252.025593484" watchObservedRunningTime="2026-07-28 00:10:09.229565708 +0000 UTC m=+3252.027530331" Jul 28 00:10:10.435412 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:10.435372 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:10:11.261738 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:11.261711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:15.224712 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:15.224680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:32.562793 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.562762 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6"] Jul 28 00:10:32.563288 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.562991 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" podUID="6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" containerName="maas-api" containerID="cri-o://a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b" gracePeriod=30 Jul 28 00:10:32.594959 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.594936 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:32.651866 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.651770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:32.705975 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.705948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:32.807050 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.807028 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:32.924458 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.924385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls\") pod \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " Jul 28 00:10:32.924458 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.924447 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgg2m\" (UniqueName: \"kubernetes.io/projected/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-kube-api-access-kgg2m\") pod \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " Jul 28 00:10:32.924684 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.924468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-openshift-service-ca\") pod \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\" (UID: \"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219\") " Jul 28 00:10:32.924834 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.924795 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" (UID: "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:10:32.926847 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.926790 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-kube-api-access-kgg2m" (OuterVolumeSpecName: "kube-api-access-kgg2m") pod "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" (UID: "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219"). InnerVolumeSpecName "kube-api-access-kgg2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:10:32.926967 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:32.926878 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" (UID: "6fa4a0f0-5a5e-46e1-825a-143ff8f4e219"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:10:33.025809 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.025780 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-maas-api-tls\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:10:33.025809 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.025805 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgg2m\" (UniqueName: \"kubernetes.io/projected/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-kube-api-access-kgg2m\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:10:33.025809 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.025833 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219-openshift-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:10:33.353825 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.353776 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" containerID="a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b" exitCode=0 Jul 28 00:10:33.353970 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.353847 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" Jul 28 00:10:33.353970 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.353871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" event={"ID":"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219","Type":"ContainerDied","Data":"a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b"} Jul 28 00:10:33.353970 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.353934 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6" event={"ID":"6fa4a0f0-5a5e-46e1-825a-143ff8f4e219","Type":"ContainerDied","Data":"7f142ba792ad9270d1020e4f550a8477c0dac8448d21f8b67849309198f01340"} Jul 28 00:10:33.353970 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.353957 2564 scope.go:117] "RemoveContainer" containerID="a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b" Jul 28 00:10:33.363693 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.363675 2564 scope.go:117] "RemoveContainer" containerID="a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b" Jul 28 00:10:33.363968 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:10:33.363950 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b\": container with ID starting with a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b not found: ID does not exist" containerID="a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b" Jul 28 00:10:33.364030 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.363976 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b"} err="failed to get container status \"a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b\": rpc error: code = NotFound desc = could not find container \"a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b\": container with ID starting with a77ced628e4bd2ba37bb22f1315733bb608ad774ef013ccdf5805b638ade456b not found: ID does not exist" Jul 28 00:10:33.374252 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.374230 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6"] Jul 28 00:10:33.377427 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.377405 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a35fc9-7476cf6f89-5wkw6"] Jul 28 00:10:33.782770 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:33.782736 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" path="/var/lib/kubelet/pods/6fa4a0f0-5a5e-46e1-825a-143ff8f4e219/volumes" Jul 28 00:10:37.145195 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:37.145162 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:10:37.190966 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:37.190938 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:37.564928 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:37.564892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:43.360535 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:43.360499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:43.426149 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:43.426099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:43.620910 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:43.620757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:10:43.831518 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:10:43.831487 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:02.049327 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.049289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:11:02.399576 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.399457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:02.427837 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.427630 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg"] Jul 28 00:11:02.428005 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.427909 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" podUID="06096818-b72c-4535-b6fa-0c90474dcaa3" containerName="payload-processing" containerID="cri-o://4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e" gracePeriod=30 Jul 28 00:11:02.502251 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.502209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:02.604601 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.604572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:02.683011 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.682989 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:11:02.770056 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.770029 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ctv4\" (UniqueName: \"kubernetes.io/projected/06096818-b72c-4535-b6fa-0c90474dcaa3-kube-api-access-8ctv4\") pod \"06096818-b72c-4535-b6fa-0c90474dcaa3\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " Jul 28 00:11:02.770211 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.770092 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06096818-b72c-4535-b6fa-0c90474dcaa3-plugins-config-volume\") pod \"06096818-b72c-4535-b6fa-0c90474dcaa3\" (UID: \"06096818-b72c-4535-b6fa-0c90474dcaa3\") " Jul 28 00:11:02.770437 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.770415 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06096818-b72c-4535-b6fa-0c90474dcaa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06096818-b72c-4535-b6fa-0c90474dcaa3" (UID: "06096818-b72c-4535-b6fa-0c90474dcaa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:11:02.772554 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.772523 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06096818-b72c-4535-b6fa-0c90474dcaa3-kube-api-access-8ctv4" (OuterVolumeSpecName: "kube-api-access-8ctv4") pod "06096818-b72c-4535-b6fa-0c90474dcaa3" (UID: "06096818-b72c-4535-b6fa-0c90474dcaa3"). InnerVolumeSpecName "kube-api-access-8ctv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:11:02.871273 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.871229 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06096818-b72c-4535-b6fa-0c90474dcaa3-plugins-config-volume\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:11:02.871273 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:02.871267 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ctv4\" (UniqueName: \"kubernetes.io/projected/06096818-b72c-4535-b6fa-0c90474dcaa3-kube-api-access-8ctv4\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:11:03.518640 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.518611 2564 generic.go:356] "Generic (PLEG): container finished" podID="06096818-b72c-4535-b6fa-0c90474dcaa3" containerID="4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e" exitCode=0 Jul 28 00:11:03.519083 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.518658 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" Jul 28 00:11:03.519083 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.518672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" event={"ID":"06096818-b72c-4535-b6fa-0c90474dcaa3","Type":"ContainerDied","Data":"4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e"} Jul 28 00:11:03.519083 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.518714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg" event={"ID":"06096818-b72c-4535-b6fa-0c90474dcaa3","Type":"ContainerDied","Data":"c68692ff93d92e1b25750a417aafd2d2241c9e65c8e87c438b18ace75acf1a56"} Jul 28 00:11:03.519083 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.518736 2564 scope.go:117] "RemoveContainer" containerID="4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e" Jul 28 00:11:03.530919 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.530843 2564 scope.go:117] "RemoveContainer" containerID="4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e" Jul 28 00:11:03.531279 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:11:03.531255 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e\": container with ID starting with 4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e not found: ID does not exist" containerID="4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e" Jul 28 00:11:03.531349 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.531287 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e"} err="failed to get container status \"4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e\": rpc error: code = NotFound desc = could not find container \"4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e\": container with ID starting with 4485ce9038107a171a7a43596d429bcb8a1869f598c51ac85e3884617deb705e not found: ID does not exist" Jul 28 00:11:03.541289 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.541267 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg"] Jul 28 00:11:03.543358 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.543339 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4a6d94-645d76c7db-jjzcg"] Jul 28 00:11:03.783260 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:03.783188 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06096818-b72c-4535-b6fa-0c90474dcaa3" path="/var/lib/kubelet/pods/06096818-b72c-4535-b6fa-0c90474dcaa3/volumes" Jul 28 00:11:07.059611 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:07.059577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:07.438671 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:07.438587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:12.462178 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:12.462143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:13.340655 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:13.340623 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:11:29.150943 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.150911 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6742k"] Jul 28 00:11:29.151346 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.151250 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06096818-b72c-4535-b6fa-0c90474dcaa3" containerName="payload-processing" Jul 28 00:11:29.151346 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.151261 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="06096818-b72c-4535-b6fa-0c90474dcaa3" containerName="payload-processing" Jul 28 00:11:29.151346 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.151275 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" containerName="maas-api" Jul 28 00:11:29.151346 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.151281 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" containerName="maas-api" Jul 28 00:11:29.151346 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.151344 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa4a0f0-5a5e-46e1-825a-143ff8f4e219" containerName="maas-api" Jul 28 00:11:29.151531 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.151354 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="06096818-b72c-4535-b6fa-0c90474dcaa3" containerName="payload-processing" Jul 28 00:11:29.154371 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.154355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:29.156463 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.156445 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-655t8\"" Jul 28 00:11:29.161430 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.161183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6742k"] Jul 28 00:11:29.295127 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.295094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtrl\" (UniqueName: \"kubernetes.io/projected/94e9a036-d1a1-4043-b50f-b4253c1a88d4-kube-api-access-6mtrl\") pod \"kuadrant-operator-catalog-6742k\" (UID: \"94e9a036-d1a1-4043-b50f-b4253c1a88d4\") " pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:29.396358 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.396331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtrl\" (UniqueName: \"kubernetes.io/projected/94e9a036-d1a1-4043-b50f-b4253c1a88d4-kube-api-access-6mtrl\") pod \"kuadrant-operator-catalog-6742k\" (UID: \"94e9a036-d1a1-4043-b50f-b4253c1a88d4\") " pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:29.404181 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.404123 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtrl\" (UniqueName: \"kubernetes.io/projected/94e9a036-d1a1-4043-b50f-b4253c1a88d4-kube-api-access-6mtrl\") pod \"kuadrant-operator-catalog-6742k\" (UID: \"94e9a036-d1a1-4043-b50f-b4253c1a88d4\") " pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:29.464718 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.464693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:29.590531 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.590507 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6742k"] Jul 28 00:11:29.592398 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:11:29.592372 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e9a036_d1a1_4043_b50f_b4253c1a88d4.slice/crio-80485336198e5bb5a5efd6530030ae4c22ccbbfbf7c1c7043470e775e9d39998 WatchSource:0}: Error finding container 80485336198e5bb5a5efd6530030ae4c22ccbbfbf7c1c7043470e775e9d39998: Status 404 returned error can't find the container with id 80485336198e5bb5a5efd6530030ae4c22ccbbfbf7c1c7043470e775e9d39998 Jul 28 00:11:29.665763 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:29.665697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6742k" event={"ID":"94e9a036-d1a1-4043-b50f-b4253c1a88d4","Type":"ContainerStarted","Data":"80485336198e5bb5a5efd6530030ae4c22ccbbfbf7c1c7043470e775e9d39998"} Jul 28 00:11:31.680199 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:31.679579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6742k" event={"ID":"94e9a036-d1a1-4043-b50f-b4253c1a88d4","Type":"ContainerStarted","Data":"c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463"} Jul 28 00:11:31.696980 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:31.696932 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6742k" podStartSLOduration=0.752287057 podStartE2EDuration="2.69691901s" totalImagesPullingTime="1.944631953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:11:31.695324111 +0000 UTC m=+3334.493288734" watchObservedRunningTime="2026-07-28 00:11:31.69691901 +0000 UTC m=+3334.494883634" Jul 28 00:11:39.465232 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:39.465179 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:39.465740 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:39.465413 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:39.486971 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:39.486951 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:39.755857 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:39.755808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:39.785648 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:39.785619 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6742k"] Jul 28 00:11:41.745224 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:41.745189 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6742k" podUID="94e9a036-d1a1-4043-b50f-b4253c1a88d4" containerName="registry-server" containerID="cri-o://c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463" gracePeriod=2 Jul 28 00:11:41.998250 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:41.998188 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:42.109412 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.109380 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mtrl\" (UniqueName: \"kubernetes.io/projected/94e9a036-d1a1-4043-b50f-b4253c1a88d4-kube-api-access-6mtrl\") pod \"94e9a036-d1a1-4043-b50f-b4253c1a88d4\" (UID: \"94e9a036-d1a1-4043-b50f-b4253c1a88d4\") " Jul 28 00:11:42.111489 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.111467 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e9a036-d1a1-4043-b50f-b4253c1a88d4-kube-api-access-6mtrl" (OuterVolumeSpecName: "kube-api-access-6mtrl") pod "94e9a036-d1a1-4043-b50f-b4253c1a88d4" (UID: "94e9a036-d1a1-4043-b50f-b4253c1a88d4"). InnerVolumeSpecName "kube-api-access-6mtrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:11:42.210098 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.210064 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mtrl\" (UniqueName: \"kubernetes.io/projected/94e9a036-d1a1-4043-b50f-b4253c1a88d4-kube-api-access-6mtrl\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:11:42.751927 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.751884 2564 generic.go:356] "Generic (PLEG): container finished" podID="94e9a036-d1a1-4043-b50f-b4253c1a88d4" containerID="c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463" exitCode=0 Jul 28 00:11:42.752374 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.751933 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6742k" Jul 28 00:11:42.752374 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.751941 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6742k" event={"ID":"94e9a036-d1a1-4043-b50f-b4253c1a88d4","Type":"ContainerDied","Data":"c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463"} Jul 28 00:11:42.752374 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.751993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6742k" event={"ID":"94e9a036-d1a1-4043-b50f-b4253c1a88d4","Type":"ContainerDied","Data":"80485336198e5bb5a5efd6530030ae4c22ccbbfbf7c1c7043470e775e9d39998"} Jul 28 00:11:42.752374 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.752019 2564 scope.go:117] "RemoveContainer" containerID="c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463" Jul 28 00:11:42.761648 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.761630 2564 scope.go:117] "RemoveContainer" containerID="c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463" Jul 28 00:11:42.761931 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:11:42.761891 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463\": container with ID starting with c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463 not found: ID does not exist" containerID="c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463" Jul 28 00:11:42.761979 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.761941 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463"} err="failed to get container status \"c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463\": rpc error: code = NotFound desc = could not find container \"c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463\": container with ID starting with c21fdbcf3fa07a3984c294301c3094d4767e60fbdad27c5725ba82dfbd5f1463 not found: ID does not exist" Jul 28 00:11:42.771601 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.771574 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6742k"] Jul 28 00:11:42.775006 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:42.774985 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6742k"] Jul 28 00:11:43.782602 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:43.782568 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e9a036-d1a1-4043-b50f-b4253c1a88d4" path="/var/lib/kubelet/pods/94e9a036-d1a1-4043-b50f-b4253c1a88d4/volumes" Jul 28 00:11:49.090203 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:49.090171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:50.217184 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:50.217154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:54.826420 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:54.826387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:54.926956 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:54.926928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:55.104849 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:55.104752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:11:55.340132 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:11:55.340092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:05.810332 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.810295 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd"] Jul 28 00:12:05.810918 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.810553 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" podUID="ecc0d4c2-150f-4449-a238-6e6f831d1fc6" containerName="maas-api" containerID="cri-o://86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1" gracePeriod=30 Jul 28 00:12:05.840215 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.840184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:05.871350 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.871321 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc"] Jul 28 00:12:05.871607 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.871581 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" podUID="5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" containerName="payload-processing" containerID="cri-o://f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264" gracePeriod=30 Jul 28 00:12:05.877614 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.877589 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb"] Jul 28 00:12:05.877867 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.877789 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" podUID="2a20d1ba-180f-4884-85bf-c2148ee2bfe4" containerName="payload-pre-processing" containerID="cri-o://b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627" gracePeriod=30 Jul 28 00:12:05.904795 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.904768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:05.957359 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:05.957334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:06.136363 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.136329 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:12:06.159580 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.159548 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:12:06.169416 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.169393 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:12:06.212891 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.212861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-plugins-config-volume\") pod \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " Jul 28 00:12:06.213051 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.212914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-maas-api-tls\") pod \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " Jul 28 00:12:06.213051 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.212942 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-plugins-config-volume\") pod \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " Jul 28 00:12:06.213051 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.212968 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-kube-api-access-rnsgs\") pod \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " Jul 28 00:12:06.213051 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.212987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr77f\" (UniqueName: \"kubernetes.io/projected/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-kube-api-access-tr77f\") pod \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\" (UID: \"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27\") " Jul 28 00:12:06.213051 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.213009 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-kube-api-access-bth24\") pod \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\" (UID: \"2a20d1ba-180f-4884-85bf-c2148ee2bfe4\") " Jul 28 00:12:06.213317 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.213087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-openshift-service-ca\") pod \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\" (UID: \"ecc0d4c2-150f-4449-a238-6e6f831d1fc6\") " Jul 28 00:12:06.213317 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.213273 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" (UID: "5fb5fbbf-8eb7-49c7-8772-43a7f0726c27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:12:06.213435 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.213359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a20d1ba-180f-4884-85bf-c2148ee2bfe4" (UID: "2a20d1ba-180f-4884-85bf-c2148ee2bfe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:12:06.213435 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.213406 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-plugins-config-volume\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.213620 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.213592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ecc0d4c2-150f-4449-a238-6e6f831d1fc6" (UID: "ecc0d4c2-150f-4449-a238-6e6f831d1fc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:12:06.215362 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.215331 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ecc0d4c2-150f-4449-a238-6e6f831d1fc6" (UID: "ecc0d4c2-150f-4449-a238-6e6f831d1fc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:12:06.215362 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.215346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-kube-api-access-tr77f" (OuterVolumeSpecName: "kube-api-access-tr77f") pod "5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" (UID: "5fb5fbbf-8eb7-49c7-8772-43a7f0726c27"). InnerVolumeSpecName "kube-api-access-tr77f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:12:06.215362 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.215356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-kube-api-access-rnsgs" (OuterVolumeSpecName: "kube-api-access-rnsgs") pod "ecc0d4c2-150f-4449-a238-6e6f831d1fc6" (UID: "ecc0d4c2-150f-4449-a238-6e6f831d1fc6"). InnerVolumeSpecName "kube-api-access-rnsgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:12:06.215527 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.215378 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-kube-api-access-bth24" (OuterVolumeSpecName: "kube-api-access-bth24") pod "2a20d1ba-180f-4884-85bf-c2148ee2bfe4" (UID: "2a20d1ba-180f-4884-85bf-c2148ee2bfe4"). InnerVolumeSpecName "kube-api-access-bth24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:12:06.314663 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.314624 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-plugins-config-volume\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.314663 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.314656 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnsgs\" (UniqueName: \"kubernetes.io/projected/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-kube-api-access-rnsgs\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.314663 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.314665 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr77f\" (UniqueName: \"kubernetes.io/projected/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27-kube-api-access-tr77f\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.314663 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.314673 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bth24\" (UniqueName: \"kubernetes.io/projected/2a20d1ba-180f-4884-85bf-c2148ee2bfe4-kube-api-access-bth24\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.314989 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.314684 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-openshift-service-ca\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.314989 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.314692 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc0d4c2-150f-4449-a238-6e6f831d1fc6-maas-api-tls\") on node \"ip-10-0-141-248.ec2.internal\" DevicePath \"\"" Jul 28 00:12:06.895924 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.895884 2564 generic.go:356] "Generic (PLEG): container finished" podID="2a20d1ba-180f-4884-85bf-c2148ee2bfe4" containerID="b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627" exitCode=0 Jul 28 00:12:06.896299 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.895928 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" Jul 28 00:12:06.896299 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.895972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" event={"ID":"2a20d1ba-180f-4884-85bf-c2148ee2bfe4","Type":"ContainerDied","Data":"b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627"} Jul 28 00:12:06.896299 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.896029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb" event={"ID":"2a20d1ba-180f-4884-85bf-c2148ee2bfe4","Type":"ContainerDied","Data":"4f6e36e1037ddea0254ec592e6d30b8f58ee00c5ef746e3b5b6169026831bf8d"} Jul 28 00:12:06.896299 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.896064 2564 scope.go:117] "RemoveContainer" containerID="b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627" Jul 28 00:12:06.897549 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.897310 2564 generic.go:356] "Generic (PLEG): container finished" podID="5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" containerID="f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264" exitCode=0 Jul 28 00:12:06.897549 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.897366 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" Jul 28 00:12:06.897549 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.897395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" event={"ID":"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27","Type":"ContainerDied","Data":"f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264"} Jul 28 00:12:06.897549 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.897458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc" event={"ID":"5fb5fbbf-8eb7-49c7-8772-43a7f0726c27","Type":"ContainerDied","Data":"592cf31b1508dce6050e9419d7e2234253445f41c45b39676bf561346dbf1e1a"} Jul 28 00:12:06.899275 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.899249 2564 generic.go:356] "Generic (PLEG): container finished" podID="ecc0d4c2-150f-4449-a238-6e6f831d1fc6" containerID="86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1" exitCode=0 Jul 28 00:12:06.899415 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.899281 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" Jul 28 00:12:06.899415 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.899342 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" event={"ID":"ecc0d4c2-150f-4449-a238-6e6f831d1fc6","Type":"ContainerDied","Data":"86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1"} Jul 28 00:12:06.899415 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.899398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd" event={"ID":"ecc0d4c2-150f-4449-a238-6e6f831d1fc6","Type":"ContainerDied","Data":"45e1855ccce72e7c7326e098a6c083d4ae854ce3fbd7d6e29d577185097ae9bc"} Jul 28 00:12:06.907476 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.907459 2564 scope.go:117] "RemoveContainer" containerID="b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627" Jul 28 00:12:06.907895 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:12:06.907872 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627\": container with ID starting with b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627 not found: ID does not exist" containerID="b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627" Jul 28 00:12:06.907985 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.907905 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627"} err="failed to get container status \"b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627\": rpc error: code = NotFound desc = could not find container \"b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627\": container with ID starting with b0267102077e42ca7b30ef0d3407bb11c65d5ef2beedd96286c18107401a9627 not found: ID does not exist" Jul 28 00:12:06.907985 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.907922 2564 scope.go:117] "RemoveContainer" containerID="f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264" Jul 28 00:12:06.916691 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.916674 2564 scope.go:117] "RemoveContainer" containerID="f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264" Jul 28 00:12:06.916995 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:12:06.916970 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264\": container with ID starting with f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264 not found: ID does not exist" containerID="f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264" Jul 28 00:12:06.917094 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.917001 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264"} err="failed to get container status \"f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264\": rpc error: code = NotFound desc = could not find container \"f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264\": container with ID starting with f1d1884d07b1a9ab09d5928821304df766b19e4032adeb22a3b7da4212d93264 not found: ID does not exist" Jul 28 00:12:06.917094 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.917018 2564 scope.go:117] "RemoveContainer" containerID="86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1" Jul 28 00:12:06.919163 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.919140 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb"] Jul 28 00:12:06.925236 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.925148 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87c90a-6568b68664-pnbcb"] Jul 28 00:12:06.927806 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.927789 2564 scope.go:117] "RemoveContainer" containerID="86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1" Jul 28 00:12:06.928120 ip-10-0-141-248 kubenswrapper[2564]: E0728 00:12:06.928102 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1\": container with ID starting with 86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1 not found: ID does not exist" containerID="86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1" Jul 28 00:12:06.928178 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.928128 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1"} err="failed to get container status \"86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1\": rpc error: code = NotFound desc = could not find container \"86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1\": container with ID starting with 86e78aadc43effa8e4389e9adff2da3a637eebb41b8282b27878cd1c9642fda1 not found: ID does not exist" Jul 28 00:12:06.936373 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.936353 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc"] Jul 28 00:12:06.941569 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.941550 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87c90a-7dbf87998d-v84dc"] Jul 28 00:12:06.950033 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.950014 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd"] Jul 28 00:12:06.953545 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:06.953522 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-87c90a-84bc996c58-brcpd"] Jul 28 00:12:07.782141 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:07.782112 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a20d1ba-180f-4884-85bf-c2148ee2bfe4" path="/var/lib/kubelet/pods/2a20d1ba-180f-4884-85bf-c2148ee2bfe4/volumes" Jul 28 00:12:07.782460 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:07.782447 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" path="/var/lib/kubelet/pods/5fb5fbbf-8eb7-49c7-8772-43a7f0726c27/volumes" Jul 28 00:12:07.782781 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:07.782769 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc0d4c2-150f-4449-a238-6e6f831d1fc6" path="/var/lib/kubelet/pods/ecc0d4c2-150f-4449-a238-6e6f831d1fc6/volumes" Jul 28 00:12:09.537991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:09.537959 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:12:10.566376 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:10.566335 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:10.920052 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:10.919966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:16.705608 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:16.705544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:16.811765 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:16.811711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:16.970911 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:16.970782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:17.177403 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:17.177366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:27.797512 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:27.797475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:27.867364 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:27.867328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:27.925175 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:27.925143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:30.745382 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:30.745343 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:12:32.374219 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:32.374182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:32.742089 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:32.741985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:38.560437 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:38.560403 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:38.630633 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:38.630596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:38.807238 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:38.807201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:39.017124 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:39.017092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:49.522953 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:49.522751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:49.622216 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:49.622178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:49.724183 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:49.724152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:52.939239 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:52.939201 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-twx6m"] Jul 28 00:12:54.199302 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:54.199267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:12:54.560996 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:12:54.560960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:00.495196 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:00.495156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:00.551098 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:00.551062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:00.738237 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:00.738200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:00.958838 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:00.958780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:01.353495 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:01.353456 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77fc7d94d5-vx5gf_78af2b2a-0f01-43cf-8ca2-24aee613bf41/authorino/0.log" Jul 28 00:13:07.238454 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:07.238426 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-xjg9v_12d0bafa-ce6a-4855-8e0c-21fd85b18ed8/manager/0.log" Jul 28 00:13:07.244518 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:07.244493 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-xjg9v_12d0bafa-ce6a-4855-8e0c-21fd85b18ed8/copy-manifests/0.log" Jul 28 00:13:07.675612 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:07.675585 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753250-czjrh_c87c96d7-5208-4774-bc0f-e3ddeaa4c2bb/cleanup/0.log" Jul 28 00:13:08.707600 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.707575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6_d2e9575e-9b0b-4de2-a599-412468b90a31/util/0.log" Jul 28 00:13:08.713256 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.713236 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6_d2e9575e-9b0b-4de2-a599-412468b90a31/pull/0.log" Jul 28 00:13:08.719501 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.719481 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6_d2e9575e-9b0b-4de2-a599-412468b90a31/extract/0.log" Jul 28 00:13:08.822700 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.822664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8_87ed1365-7594-4eee-a8b4-f4e597ceb656/util/0.log" Jul 28 00:13:08.828198 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.828174 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8_87ed1365-7594-4eee-a8b4-f4e597ceb656/pull/0.log" Jul 28 00:13:08.833674 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.833656 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8_87ed1365-7594-4eee-a8b4-f4e597ceb656/extract/0.log" Jul 28 00:13:08.946117 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.946083 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579_a9895218-e942-4183-be21-8ac1ea0c4a67/util/0.log" Jul 28 00:13:08.951728 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.951707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579_a9895218-e942-4183-be21-8ac1ea0c4a67/pull/0.log" Jul 28 00:13:08.957039 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:08.957018 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579_a9895218-e942-4183-be21-8ac1ea0c4a67/extract/0.log" Jul 28 00:13:09.068243 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:09.068212 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7_5cff4401-e346-4a76-a007-cb4f72e63dce/util/0.log" Jul 28 00:13:09.073626 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:09.073604 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7_5cff4401-e346-4a76-a007-cb4f72e63dce/pull/0.log" Jul 28 00:13:09.079075 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:09.079056 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7_5cff4401-e346-4a76-a007-cb4f72e63dce/extract/0.log" Jul 28 00:13:09.206202 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:09.206171 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77fc7d94d5-vx5gf_78af2b2a-0f01-43cf-8ca2-24aee613bf41/authorino/0.log" Jul 28 00:13:09.536177 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:09.536152 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qxbhq_c1ab9c4f-0c36-4cbe-837f-1b3208c69a75/kuadrant-console-plugin/0.log" Jul 28 00:13:09.863201 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:09.863121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-twx6m_6e8925ee-e527-48ae-bf8a-5ec8356053a9/limitador/0.log" Jul 28 00:13:10.618681 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:10.618650 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-jp7t2_54845aaf-3348-425d-b57e-3b1684c5216c/kube-auth-proxy/0.log" Jul 28 00:13:11.068297 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:11.068264 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74f9f647d5-fbgwn_26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5/router/0.log" Jul 28 00:13:11.812440 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:11.812410 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4mkqr_a2339274-a34d-4b6f-8f73-48430893112b/main/0.log" Jul 28 00:13:16.737530 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737497 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lrsk/must-gather-8vgvk"] Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737838 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e9a036-d1a1-4043-b50f-b4253c1a88d4" containerName="registry-server" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737849 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e9a036-d1a1-4043-b50f-b4253c1a88d4" containerName="registry-server" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737867 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" containerName="payload-processing" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737873 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" containerName="payload-processing" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737889 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a20d1ba-180f-4884-85bf-c2148ee2bfe4" containerName="payload-pre-processing" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737898 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a20d1ba-180f-4884-85bf-c2148ee2bfe4" containerName="payload-pre-processing" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737907 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc0d4c2-150f-4449-a238-6e6f831d1fc6" containerName="maas-api" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737914 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc0d4c2-150f-4449-a238-6e6f831d1fc6" containerName="maas-api" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737977 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a20d1ba-180f-4884-85bf-c2148ee2bfe4" containerName="payload-pre-processing" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737986 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb5fbbf-8eb7-49c7-8772-43a7f0726c27" containerName="payload-processing" Jul 28 00:13:16.737991 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737992 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e9a036-d1a1-4043-b50f-b4253c1a88d4" containerName="registry-server" Jul 28 00:13:16.738377 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.737999 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc0d4c2-150f-4449-a238-6e6f831d1fc6" containerName="maas-api" Jul 28 00:13:16.741223 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.741204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:16.743708 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.743687 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lrsk\"/\"openshift-service-ca.crt\"" Jul 28 00:13:16.743853 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.743723 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9lrsk\"/\"kube-root-ca.crt\"" Jul 28 00:13:16.749076 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.749056 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/must-gather-8vgvk"] Jul 28 00:13:16.753124 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.753105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:16.833896 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.833865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgzm\" (UniqueName: \"kubernetes.io/projected/6ceea210-5ee3-44b0-af6d-bac7ef81f6df-kube-api-access-xkgzm\") pod \"must-gather-8vgvk\" (UID: \"6ceea210-5ee3-44b0-af6d-bac7ef81f6df\") " pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:16.833896 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.833898 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ceea210-5ee3-44b0-af6d-bac7ef81f6df-must-gather-output\") pod \"must-gather-8vgvk\" (UID: \"6ceea210-5ee3-44b0-af6d-bac7ef81f6df\") " pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:16.852153 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.852128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" limit=1024 Jul 28 00:13:16.935238 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.935203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ceea210-5ee3-44b0-af6d-bac7ef81f6df-must-gather-output\") pod \"must-gather-8vgvk\" (UID: \"6ceea210-5ee3-44b0-af6d-bac7ef81f6df\") " pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:16.935403 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.935298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgzm\" (UniqueName: \"kubernetes.io/projected/6ceea210-5ee3-44b0-af6d-bac7ef81f6df-kube-api-access-xkgzm\") pod \"must-gather-8vgvk\" (UID: \"6ceea210-5ee3-44b0-af6d-bac7ef81f6df\") " pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:16.935578 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.935546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6ceea210-5ee3-44b0-af6d-bac7ef81f6df-must-gather-output\") pod \"must-gather-8vgvk\" (UID: \"6ceea210-5ee3-44b0-af6d-bac7ef81f6df\") " pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:16.943912 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:16.943881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgzm\" (UniqueName: \"kubernetes.io/projected/6ceea210-5ee3-44b0-af6d-bac7ef81f6df-kube-api-access-xkgzm\") pod \"must-gather-8vgvk\" (UID: \"6ceea210-5ee3-44b0-af6d-bac7ef81f6df\") " pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:17.051062 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:17.051032 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/must-gather-8vgvk" Jul 28 00:13:17.187761 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:17.187724 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/must-gather-8vgvk"] Jul 28 00:13:17.189516 ip-10-0-141-248 kubenswrapper[2564]: W0728 00:13:17.189492 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ceea210_5ee3_44b0_af6d_bac7ef81f6df.slice/crio-3ed91aa3b68e117dbb6ea5aac0d45b50f0ed897571890780aa27ec58265ea80b WatchSource:0}: Error finding container 3ed91aa3b68e117dbb6ea5aac0d45b50f0ed897571890780aa27ec58265ea80b: Status 404 returned error can't find the container with id 3ed91aa3b68e117dbb6ea5aac0d45b50f0ed897571890780aa27ec58265ea80b Jul 28 00:13:17.191308 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:17.191291 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:13:17.300280 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:17.300234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/must-gather-8vgvk" event={"ID":"6ceea210-5ee3-44b0-af6d-bac7ef81f6df","Type":"ContainerStarted","Data":"3ed91aa3b68e117dbb6ea5aac0d45b50f0ed897571890780aa27ec58265ea80b"} Jul 28 00:13:18.311248 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:18.311177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/must-gather-8vgvk" event={"ID":"6ceea210-5ee3-44b0-af6d-bac7ef81f6df","Type":"ContainerStarted","Data":"dbfa288079d0eda1f508cd37b5586919dc10c56f8d4d41a7ce2e730847108b28"} Jul 28 00:13:19.319358 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:19.319315 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/must-gather-8vgvk" event={"ID":"6ceea210-5ee3-44b0-af6d-bac7ef81f6df","Type":"ContainerStarted","Data":"84fc3fc47fa99ef2142ea1b0d74af0d998d5992e3f4f406ea1bb1ee3af37b51a"} Jul 28 00:13:19.335305 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:19.335253 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9lrsk/must-gather-8vgvk" podStartSLOduration=2.352340996 podStartE2EDuration="3.335236984s" totalImagesPullingTime="982.895988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:13:19.333211846 +0000 UTC m=+3442.131176539" watchObservedRunningTime="2026-07-28 00:13:19.335236984 +0000 UTC m=+3442.133201662" Jul 28 00:13:19.723100 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:19.723023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vw4gr_65b91810-92da-4848-b465-2025a4ce7ec0/global-pull-secret-syncer/0.log" Jul 28 00:13:19.774324 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:19.774294 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4n869_5455e7ab-9da8-45ac-8d9a-ea1f93ac7a0a/konnectivity-agent/0.log" Jul 28 00:13:19.889615 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:19.889583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-248.ec2.internal_12d7ac228f243a07d086e226310b600c/haproxy/0.log" Jul 28 00:13:24.318245 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.318136 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6_d2e9575e-9b0b-4de2-a599-412468b90a31/extract/0.log" Jul 28 00:13:24.347067 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.347041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6_d2e9575e-9b0b-4de2-a599-412468b90a31/util/0.log" Jul 28 00:13:24.369439 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.369413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592njs6_d2e9575e-9b0b-4de2-a599-412468b90a31/pull/0.log" Jul 28 00:13:24.394103 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.394078 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8_87ed1365-7594-4eee-a8b4-f4e597ceb656/extract/0.log" Jul 28 00:13:24.416828 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.416788 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8_87ed1365-7594-4eee-a8b4-f4e597ceb656/util/0.log" Jul 28 00:13:24.444620 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.444560 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e079bf8_87ed1365-7594-4eee-a8b4-f4e597ceb656/pull/0.log" Jul 28 00:13:24.476784 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.476735 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579_a9895218-e942-4183-be21-8ac1ea0c4a67/extract/0.log" Jul 28 00:13:24.502222 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.502179 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579_a9895218-e942-4183-be21-8ac1ea0c4a67/util/0.log" Jul 28 00:13:24.525941 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.525892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed736r579_a9895218-e942-4183-be21-8ac1ea0c4a67/pull/0.log" Jul 28 00:13:24.551032 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.550976 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7_5cff4401-e346-4a76-a007-cb4f72e63dce/extract/0.log" Jul 28 00:13:24.571632 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.571504 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7_5cff4401-e346-4a76-a007-cb4f72e63dce/util/0.log" Jul 28 00:13:24.601165 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:24.601132 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17qkm7_5cff4401-e346-4a76-a007-cb4f72e63dce/pull/0.log" Jul 28 00:13:25.124649 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:25.124592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77fc7d94d5-vx5gf_78af2b2a-0f01-43cf-8ca2-24aee613bf41/authorino/0.log" Jul 28 00:13:25.210890 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:25.210852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-qxbhq_c1ab9c4f-0c36-4cbe-837f-1b3208c69a75/kuadrant-console-plugin/0.log" Jul 28 00:13:25.440272 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:25.440189 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-twx6m_6e8925ee-e527-48ae-bf8a-5ec8356053a9/limitador/0.log" Jul 28 00:13:27.304629 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.304570 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-slmlm_e3fa7c20-89da-424a-9c08-7053736e0cf6/cluster-monitoring-operator/0.log" Jul 28 00:13:27.401204 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.401172 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8c94c6655-bjth8_2c339d75-f668-4f72-b9e3-d7ced388b27e/metrics-server/0.log" Jul 28 00:13:27.426006 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.425971 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-fnnkf_bc6f33c0-4bd1-42bb-be29-ebaa8af21806/monitoring-plugin/0.log" Jul 28 00:13:27.592557 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.592465 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8rpt_7ad8c858-45de-4c16-bef7-7ad8222d5832/node-exporter/0.log" Jul 28 00:13:27.614937 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.614905 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8rpt_7ad8c858-45de-4c16-bef7-7ad8222d5832/kube-rbac-proxy/0.log" Jul 28 00:13:27.636098 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.636063 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l8rpt_7ad8c858-45de-4c16-bef7-7ad8222d5832/init-textfile/0.log" Jul 28 00:13:27.893909 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.893807 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6f5rt_ab9a5b23-8877-4788-8e63-efecbac839b6/prometheus-operator/0.log" Jul 28 00:13:27.914662 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:27.914631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-6f5rt_ab9a5b23-8877-4788-8e63-efecbac839b6/kube-rbac-proxy/0.log" Jul 28 00:13:28.043287 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:28.043251 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8849666-jn92r_72a822a8-7799-4a0c-95a7-e6978f9dcc51/thanos-query/0.log" Jul 28 00:13:28.061877 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:28.061851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8849666-jn92r_72a822a8-7799-4a0c-95a7-e6978f9dcc51/kube-rbac-proxy-web/0.log" Jul 28 00:13:28.082636 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:28.082609 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8849666-jn92r_72a822a8-7799-4a0c-95a7-e6978f9dcc51/kube-rbac-proxy/0.log" Jul 28 00:13:28.108763 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:28.108731 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8849666-jn92r_72a822a8-7799-4a0c-95a7-e6978f9dcc51/prom-label-proxy/0.log" Jul 28 00:13:28.130111 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:28.130077 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8849666-jn92r_72a822a8-7799-4a0c-95a7-e6978f9dcc51/kube-rbac-proxy-rules/0.log" Jul 28 00:13:28.157930 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:28.157861 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8849666-jn92r_72a822a8-7799-4a0c-95a7-e6978f9dcc51/kube-rbac-proxy-metrics/0.log" Jul 28 00:13:29.304364 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:29.304325 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-5q72l_dcf565a0-b02a-4958-9786-02ce54f8c298/networking-console-plugin/0.log" Jul 28 00:13:30.039235 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.039197 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8"] Jul 28 00:13:30.045469 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.045441 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.048551 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.048529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9lrsk\"/\"default-dockercfg-d9m9z\"" Jul 28 00:13:30.051697 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.051671 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8"] Jul 28 00:13:30.075156 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.074872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-sys\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.075156 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.074922 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-proc\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.075156 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.074955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-lib-modules\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.075156 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.075003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdvp\" (UniqueName: \"kubernetes.io/projected/3a655d1f-2366-44bf-85b2-a876b60b36f0-kube-api-access-chdvp\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.075156 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.075066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-podres\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176065 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-podres\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176247 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-sys\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176247 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-proc\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176247 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-lib-modules\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176247 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-podres\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176247 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdvp\" (UniqueName: \"kubernetes.io/projected/3a655d1f-2366-44bf-85b2-a876b60b36f0-kube-api-access-chdvp\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176492 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-proc\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176492 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-sys\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.176492 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.176400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a655d1f-2366-44bf-85b2-a876b60b36f0-lib-modules\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.186246 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.186219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdvp\" (UniqueName: \"kubernetes.io/projected/3a655d1f-2366-44bf-85b2-a876b60b36f0-kube-api-access-chdvp\") pod \"perf-node-gather-daemonset-6rts8\" (UID: \"3a655d1f-2366-44bf-85b2-a876b60b36f0\") " pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.360134 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.360018 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:30.397737 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.397707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58598dcbf4-9kgvk_da5929e6-0cc1-4dd5-8467-f83847af5370/console/0.log" Jul 28 00:13:30.543088 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:30.543043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8"] Jul 28 00:13:31.111743 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:31.111703 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-zp4sh_12e64263-e8d2-4959-b3c4-3ad2fc7c2c20/volume-data-source-validator/0.log" Jul 28 00:13:31.424573 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:31.424475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" event={"ID":"3a655d1f-2366-44bf-85b2-a876b60b36f0","Type":"ContainerStarted","Data":"6e366d1d420c76b01b49b5ce69ccc2e99523d5530dbba548a86404dd51db30b5"} Jul 28 00:13:31.424573 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:31.424542 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" event={"ID":"3a655d1f-2366-44bf-85b2-a876b60b36f0","Type":"ContainerStarted","Data":"f1f2bdf8c7a9f401c598ee96678f26008bcf2c942446f21a9d978a3290b8782b"} Jul 28 00:13:31.425266 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:31.424622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:31.440206 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:31.440143 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" podStartSLOduration=1.440119197 podStartE2EDuration="1.440119197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:13:31.437707332 +0000 UTC m=+3454.235671958" watchObservedRunningTime="2026-07-28 00:13:31.440119197 +0000 UTC m=+3454.238083824" Jul 28 00:13:32.134475 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:32.134435 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlfmr_e3deae1b-7472-44a8-881d-26b065db3335/dns/0.log" Jul 28 00:13:32.154138 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:32.154109 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rlfmr_e3deae1b-7472-44a8-881d-26b065db3335/kube-rbac-proxy/0.log" Jul 28 00:13:32.260750 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:32.260725 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlxxm_103dec8d-2e5e-4973-b634-b28b043cfb6b/dns-node-resolver/0.log" Jul 28 00:13:32.746309 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:32.746262 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29753280-skcj7_24bcd7aa-9300-441b-b872-f5f5013abd0f/image-pruner/0.log" Jul 28 00:13:32.862968 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:32.862918 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wld4r_ff37ceb7-f0de-44f8-82a6-c34d3899c294/node-ca/0.log" Jul 28 00:13:34.068408 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:34.068375 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-jp7t2_54845aaf-3348-425d-b57e-3b1684c5216c/kube-auth-proxy/0.log" Jul 28 00:13:34.215859 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:34.215748 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74f9f647d5-fbgwn_26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5/router/0.log" Jul 28 00:13:35.015334 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:35.015308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kh2xw_75cddcee-0023-4c16-828c-2294054f6c57/serve-healthcheck-canary/0.log" Jul 28 00:13:35.495317 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:35.495240 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-7mjm7_9c23353e-81b9-4556-a3a2-33b243c63f68/insights-operator/0.log" Jul 28 00:13:35.496106 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:35.496080 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-7mjm7_9c23353e-81b9-4556-a3a2-33b243c63f68/insights-operator/1.log" Jul 28 00:13:35.517839 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:35.517783 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4whwh_b95822c6-eeb5-4ccc-860b-110f03a0d0cc/kube-rbac-proxy/0.log" Jul 28 00:13:35.538548 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:35.538517 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4whwh_b95822c6-eeb5-4ccc-860b-110f03a0d0cc/exporter/0.log" Jul 28 00:13:35.562656 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:35.562628 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4whwh_b95822c6-eeb5-4ccc-860b-110f03a0d0cc/extractor/0.log" Jul 28 00:13:37.442547 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:37.442512 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9lrsk/perf-node-gather-daemonset-6rts8" Jul 28 00:13:38.458388 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:38.458356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-xjg9v_12d0bafa-ce6a-4855-8e0c-21fd85b18ed8/manager/0.log" Jul 28 00:13:38.478342 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:38.478310 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9cf867746-xjg9v_12d0bafa-ce6a-4855-8e0c-21fd85b18ed8/copy-manifests/0.log" Jul 28 00:13:39.898114 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:39.898060 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6589476d7c-jqxb5_f2e5233f-37fe-4133-b83a-6b2679ee8358/manager/0.log" Jul 28 00:13:39.945040 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:39.945012 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jl79w_c0efdf39-72f8-4309-b24d-81f6657a451d/openshift-lws-operator/0.log" Jul 28 00:13:40.644145 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:40.644115 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-jp7t2_54845aaf-3348-425d-b57e-3b1684c5216c/kube-auth-proxy/0.log" Jul 28 00:13:40.766010 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:40.765976 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74f9f647d5-fbgwn_26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5/router/0.log" Jul 28 00:13:42.524151 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:42.524119 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5946b997db-jp7t2_54845aaf-3348-425d-b57e-3b1684c5216c/kube-auth-proxy/0.log" Jul 28 00:13:42.655087 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:42.655052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74f9f647d5-fbgwn_26ec3ab0-b39f-4241-b2b4-61e84e7b4ad5/router/0.log" Jul 28 00:13:46.214396 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.214364 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/kube-multus-additional-cni-plugins/0.log" Jul 28 00:13:46.234123 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.234100 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/egress-router-binary-copy/0.log" Jul 28 00:13:46.254176 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.254148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/cni-plugins/0.log" Jul 28 00:13:46.273889 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.273853 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/bond-cni-plugin/0.log" Jul 28 00:13:46.293254 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.293224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/routeoverride-cni/0.log" Jul 28 00:13:46.312414 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.312382 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/whereabouts-cni-bincopy/0.log" Jul 28 00:13:46.331304 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.331280 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gd2zk_d92fda41-1b32-4f06-97c5-0d509c87bb90/whereabouts-cni/0.log" Jul 28 00:13:46.683418 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.683392 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z8qh9_d301bf01-44e5-43fa-b2d9-8fccf338c712/kube-multus/0.log" Jul 28 00:13:46.744438 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.744411 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l6jq8_93e62be3-2c13-473b-b298-ae8f2dde91b8/network-metrics-daemon/0.log" Jul 28 00:13:46.764219 ip-10-0-141-248 kubenswrapper[2564]: I0728 00:13:46.764197 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l6jq8_93e62be3-2c13-473b-b298-ae8f2dde91b8/kube-rbac-proxy/0.log"