Jul 28 14:18:17.080654 ip-10-0-131-121 systemd[1]: Starting Kubernetes Kubelet... Jul 28 14:18:17.526435 ip-10-0-131-121 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 14:18:17.526435 ip-10-0-131-121 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 14:18:17.526435 ip-10-0-131-121 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 14:18:17.526435 ip-10-0-131-121 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 14:18:17.526435 ip-10-0-131-121 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 14:18:17.528626 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.528523 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 14:18:17.533773 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533745 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 14:18:17.533773 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533770 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 14:18:17.533773 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533774 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 14:18:17.533773 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533777 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533788 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533792 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533795 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533797 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533800 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533805 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533808 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533811 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533814 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533816 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533819 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533822 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533825 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533827 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533830 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533833 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533835 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533838 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533840 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 14:18:17.533899 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533843 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533845 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533848 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533850 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533853 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533855 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533858 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533860 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533866 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533871 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533874 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533877 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533879 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533882 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533885 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533888 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533891 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533893 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533896 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 14:18:17.534408 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533898 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533901 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533904 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533907 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533909 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533912 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533915 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533918 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533920 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533923 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533927 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533931 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533934 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533937 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533939 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533942 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533945 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533947 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533950 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533953 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 14:18:17.534875 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533956 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533959 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533962 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533965 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533968 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533970 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533974 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533978 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533980 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533983 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533985 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533988 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533990 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533994 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533997 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.533999 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534002 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534006 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534009 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 14:18:17.535460 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534012 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534015 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534018 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534021 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534024 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534026 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534029 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534031 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534034 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534036 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534039 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534041 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534043 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534046 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534048 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534051 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534054 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534057 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534826 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 14:18:17.535926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534835 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534839 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534842 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534845 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534848 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534851 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534853 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534858 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534862 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534865 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534868 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534871 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534873 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534876 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534879 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534882 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534885 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534888 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534891 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534893 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 14:18:17.536403 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534896 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534899 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534902 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534906 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534908 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534911 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534914 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534917 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534921 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534924 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534932 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534935 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534939 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534942 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534944 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534947 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534950 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534953 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534955 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 14:18:17.536908 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534958 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534962 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534966 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534968 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534971 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534984 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534987 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534990 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534992 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534995 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.534998 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535000 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535003 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535005 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535008 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535012 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535014 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535017 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535020 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535022 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 14:18:17.537388 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535025 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535027 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535030 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535032 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535036 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535040 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535043 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535045 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535048 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535051 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535054 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535056 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535059 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535062 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535064 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535067 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535069 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535072 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535075 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535077 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 14:18:17.537868 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535080 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535083 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535086 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535088 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535091 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535093 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535096 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535099 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535102 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535104 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535107 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535109 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535112 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535114 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535117 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535120 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535122 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535125 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535131 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 14:18:17.538392 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535598 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535604 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535607 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535609 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535612 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535615 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535619 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535622 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535625 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535628 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535631 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535634 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535636 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535640 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535644 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535647 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535650 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535653 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535656 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 14:18:17.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535659 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535662 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535664 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535667 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535669 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535672 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535675 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535677 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535680 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535682 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535685 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535694 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535699 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535702 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535706 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535709 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535711 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535715 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535718 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 14:18:17.539345 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535721 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535723 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535726 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535728 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535731 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535734 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535736 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535739 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535741 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535747 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535750 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535753 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535755 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535758 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535761 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535764 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535767 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535769 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535772 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535774 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 14:18:17.539812 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535777 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535780 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535782 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535785 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535788 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535797 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535800 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535803 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535806 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535808 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535811 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535818 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535820 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535823 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535825 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535828 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535831 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535833 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535837 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535839 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 14:18:17.540458 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535842 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535845 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535847 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535850 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535852 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535855 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535857 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535860 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535863 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535865 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535867 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535870 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535872 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535875 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535877 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535880 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535882 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535887 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535890 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535892 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.535895 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 14:18:17.540942 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.535980 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.535988 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536000 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536006 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536010 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536013 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536017 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536022 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536025 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536028 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536033 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536036 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536039 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536042 2562 flags.go:64] FLAG: --cgroup-root="" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536045 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536048 2562 flags.go:64] FLAG: --client-ca-file="" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536051 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536054 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536061 2562 flags.go:64] FLAG: --cluster-domain="" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536064 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536067 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536070 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536074 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536077 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 14:18:17.541476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536080 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536083 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536086 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536089 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536093 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536096 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536105 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536109 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536113 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536116 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536119 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536122 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536125 2562 flags.go:64] FLAG: --enable-server="true" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536128 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536135 2562 flags.go:64] FLAG: --event-burst="100" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536138 2562 flags.go:64] FLAG: --event-qps="50" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536141 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536144 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536147 2562 flags.go:64] FLAG: --eviction-hard="" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536151 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536154 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536157 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536159 2562 flags.go:64] FLAG: --eviction-soft="" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536162 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536165 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 14:18:17.542060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536168 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536171 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536174 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536177 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536180 2562 flags.go:64] FLAG: --feature-gates="" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536184 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536187 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536190 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536193 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536196 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536199 2562 flags.go:64] FLAG: --help="false" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536202 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536205 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536208 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536217 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536221 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536225 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536228 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536231 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536234 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536236 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536239 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536242 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536245 2562 flags.go:64] FLAG: --kube-reserved="" Jul 28 14:18:17.542703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536248 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536251 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536254 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536257 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536259 2562 flags.go:64] FLAG: --lock-file="" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536262 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536265 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536268 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536286 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536289 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536292 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536295 2562 flags.go:64] FLAG: --logging-format="text" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536298 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536302 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536304 2562 flags.go:64] FLAG: --manifest-url="" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536308 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536314 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536317 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536328 2562 flags.go:64] FLAG: --max-pods="110" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536331 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536334 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536338 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536341 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536345 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536348 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 14:18:17.543308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536351 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536361 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536364 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536367 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536370 2562 flags.go:64] FLAG: --pod-cidr="" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536373 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536380 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536383 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536387 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536390 2562 flags.go:64] FLAG: --port="10250" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536393 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536396 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-013796a439ebf6d60" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536399 2562 flags.go:64] FLAG: --qos-reserved="" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536402 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536406 2562 flags.go:64] FLAG: --register-node="true" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536411 2562 flags.go:64] FLAG: --register-with-taints="" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536415 2562 flags.go:64] FLAG: --registry-burst="10" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536418 2562 flags.go:64] FLAG: --registry-qps="5" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536421 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536424 2562 flags.go:64] FLAG: --reserved-memory="" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536428 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536431 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536434 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536437 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536440 2562 flags.go:64] FLAG: --runonce="false" Jul 28 14:18:17.544001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536443 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536447 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536450 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536453 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536456 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536459 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536463 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536467 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536469 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536472 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536475 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536478 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536481 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536485 2562 flags.go:64] FLAG: --system-cgroups="" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536488 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536493 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536496 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536499 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536506 2562 flags.go:64] FLAG: --tls-min-version="" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536508 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536511 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536514 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536519 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536522 2562 flags.go:64] FLAG: --v="2" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536526 2562 flags.go:64] FLAG: --version="false" Jul 28 14:18:17.544699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536530 2562 flags.go:64] FLAG: --vmodule="" Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536534 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.536537 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536668 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536672 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536675 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536679 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536682 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536685 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536689 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536691 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536694 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536697 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536700 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536703 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536706 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536708 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536711 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536714 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536716 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 14:18:17.545384 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536719 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536723 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536727 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536730 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536733 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536736 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536739 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536742 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536745 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536750 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536753 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536756 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536758 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536761 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536764 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536767 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536769 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536772 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536774 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 14:18:17.545926 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536777 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536780 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536785 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536788 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536791 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536793 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536796 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536798 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536802 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536804 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536807 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536810 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536813 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536815 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536818 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536820 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536824 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536826 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536829 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 14:18:17.546683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536832 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536834 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536837 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536841 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536844 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536846 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536849 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536852 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536855 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536857 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536860 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536862 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536865 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536868 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536870 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536875 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536877 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536880 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536882 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536885 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 14:18:17.547353 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536889 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536893 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536898 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536901 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536904 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536907 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536910 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536913 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536917 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536921 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536924 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536927 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536930 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536934 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536936 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536940 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536943 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536946 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536948 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 14:18:17.547856 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536951 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536954 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536957 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536959 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.536962 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.537883 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.547869 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 14:18:17.548338 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.547889 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 14:18:17.548882 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.548869 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 14:18:17.548917 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.548885 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 14:18:17.548962 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548953 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 14:18:17.548962 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548962 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548966 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548969 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548973 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548976 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548978 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548981 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548984 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548986 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548989 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548992 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548995 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.548998 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549001 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549003 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549006 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549009 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549012 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549016 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 14:18:17.549016 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549019 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549022 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549024 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549028 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549030 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549033 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549036 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549039 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549042 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549044 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549047 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549050 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549053 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549055 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549058 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549061 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549063 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549066 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549068 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549071 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 14:18:17.549640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549073 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549076 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549078 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549081 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549083 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549086 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549091 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549093 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549096 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549100 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549104 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549107 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549110 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549112 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549117 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549120 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549123 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549126 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549128 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 14:18:17.550126 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549133 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549136 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549139 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549142 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549145 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549148 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549150 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549153 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549156 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549158 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549161 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549163 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549166 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549168 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549171 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549173 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549176 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549179 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549182 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549185 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 14:18:17.550640 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549188 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549191 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549194 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549197 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549200 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549202 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549205 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549208 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549212 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549215 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549219 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549222 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549225 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549227 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549230 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549233 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549236 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549238 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549241 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 14:18:17.551138 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549244 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.549250 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549423 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549431 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549434 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549437 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549440 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549443 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549446 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549448 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549452 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549454 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549457 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549460 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549465 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 14:18:17.551629 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549468 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549471 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549473 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549476 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549479 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549482 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549484 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549487 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549490 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549492 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549495 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549498 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549501 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549503 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549506 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549508 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549511 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549513 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549516 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 14:18:17.551997 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549518 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549521 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549523 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549526 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549528 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549531 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549533 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549536 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549539 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549542 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549544 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549547 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549551 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549555 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549558 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549561 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549563 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549566 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549569 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 14:18:17.552476 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549572 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549574 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549579 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549582 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549585 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549588 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549591 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549594 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549597 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549599 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549602 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549604 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549607 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549609 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549612 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549614 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549617 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549619 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549622 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 14:18:17.552942 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549624 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549627 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549630 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549633 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549635 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549638 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549641 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549643 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549646 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549649 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549652 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549654 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549657 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549660 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549663 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549665 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549668 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549671 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549673 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549676 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549679 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 14:18:17.553520 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549681 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549684 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549686 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549689 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549693 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549696 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549699 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:17.549701 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.549707 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 14:18:17.554062 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.549865 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 14:18:17.554861 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.554843 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 14:18:17.556603 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.556590 2562 server.go:1019] "Starting client certificate rotation" Jul 28 14:18:17.556722 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.556700 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 14:18:17.556759 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.556749 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 14:18:17.559518 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.559501 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 14:18:17.563635 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.563612 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 14:18:17.575069 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.575040 2562 log.go:25] "Validated CRI v1 runtime API" Jul 28 14:18:17.577897 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.577876 2562 log.go:25] "Validated CRI v1 image API" Jul 28 14:18:17.578731 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.578710 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 14:18:17.582263 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.582230 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b2f6bd9-9a8d-4cc5-b5f3-cdbb9ad81c87:/dev/nvme0n1p3 d42e70ea-7a96-4a18-af7c-7c815a1eb533:/dev/nvme0n1p4] Jul 28 14:18:17.582340 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.582262 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 14:18:17.588056 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.587929 2562 manager.go:216] Machine: {Timestamp:2026-07-28 14:18:17.587427043 +0000 UTC m=+0.375067644 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100258 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a31dd2cce7e45058f358b20272fe SystemUUID:ec27a31d-d2cc-e7e4-5058-f358b20272fe BootID:afe91a39-a435-4e27-b154-7ca233717e60 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:cf:91:7e:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:cf:91:7e:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:4c:a8:7e:18:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 14:18:17.588056 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.588047 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 14:18:17.588167 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.588142 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 14:18:17.589239 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.589222 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 14:18:17.591195 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.591163 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 14:18:17.591361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.591202 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 14:18:17.592925 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.592911 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 14:18:17.592962 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.592928 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 14:18:17.592962 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.592936 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 14:18:17.593759 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.593745 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 14:18:17.595096 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.595081 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 14:18:17.598639 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598618 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 28 14:18:17.598786 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598775 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 14:18:17.598832 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598810 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 28 14:18:17.598832 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598823 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 14:18:17.598884 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598836 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 14:18:17.598884 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598845 2562 kubelet.go:398] "Adding apiserver pod source" Jul 28 14:18:17.598884 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.598856 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 14:18:17.599337 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.599325 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 14:18:17.599380 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.599347 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 14:18:17.601785 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.601759 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 14:18:17.604153 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.604138 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 14:18:17.604222 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.604170 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 14:18:17.606372 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606358 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606376 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606382 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606388 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606397 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606403 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606411 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606417 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 14:18:17.606422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606425 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 14:18:17.606636 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606431 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 14:18:17.606636 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606440 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 14:18:17.606636 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.606446 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 14:18:17.608078 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.608068 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 14:18:17.608078 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.608077 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 14:18:17.610920 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.610894 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 14:18:17.611034 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.610928 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 14:18:17.611034 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.610939 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 14:18:17.611481 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.611468 2562 server.go:1268] "Started kubelet" Jul 28 14:18:17.611573 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.611543 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 14:18:17.611619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.611563 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 14:18:17.611672 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.611642 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 14:18:17.612344 ip-10-0-131-121 systemd[1]: Started Kubernetes Kubelet. Jul 28 14:18:17.613877 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.613857 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 14:18:17.614390 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.614377 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 28 14:18:17.618865 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.618845 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 14:18:17.623460 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.623444 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 14:18:17.623532 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.623479 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 14:18:17.623594 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.623532 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:17.624416 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.624399 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 14:18:17.624416 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.624417 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 14:18:17.624630 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.624539 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 14:18:17.624630 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.624586 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 28 14:18:17.624630 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.624596 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 28 14:18:17.625887 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.625868 2562 factory.go:55] Registering systemd factory Jul 28 14:18:17.625887 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.625888 2562 factory.go:223] Registration of the systemd container factory successfully Jul 28 14:18:17.626228 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.626212 2562 factory.go:153] Registering CRI-O factory Jul 28 14:18:17.626341 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.626329 2562 factory.go:223] Registration of the crio container factory successfully Jul 28 14:18:17.626413 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.626389 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 14:18:17.626413 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.626395 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 14:18:17.626505 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.626416 2562 factory.go:103] Registering Raw factory Jul 28 14:18:17.626505 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.626435 2562 manager.go:1196] Started watching for new ooms in manager Jul 28 14:18:17.626505 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.626254 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 14:18:17.626862 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.626850 2562 manager.go:349] Starting recovery of all containers Jul 28 14:18:17.627453 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.626231 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c679de37002f6b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-28 14:18:17.611439979 +0000 UTC m=+0.399080579,LastTimestamp:2026-07-28 14:18:17.611439979 +0000 UTC m=+0.399080579,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}" Jul 28 14:18:17.639648 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.639622 2562 manager.go:354] Recovery completed Jul 28 14:18:17.640816 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.640795 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vgn7g" Jul 28 14:18:17.644425 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.644347 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.647497 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.647470 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.647607 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.647495 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vgn7g" Jul 28 14:18:17.647607 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.647507 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.647607 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.647550 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.648211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.648197 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 14:18:17.648211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.648210 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 14:18:17.648322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.648226 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 28 14:18:17.648322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.648227 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.649070 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.649046 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.649070 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.649071 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.649162 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.649086 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.649763 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.649678 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-121.ec2.internal.18c679de392652ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-121.ec2.internal,UID:ip-10-0-131-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-121.ec2.internal,},FirstTimestamp:2026-07-28 14:18:17.647493871 +0000 UTC m=+0.435134475,LastTimestamp:2026-07-28 14:18:17.647493871 +0000 UTC m=+0.435134475,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-121.ec2.internal,}" Jul 28 14:18:17.652579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.652560 2562 policy_none.go:49] "None policy: Start" Jul 28 14:18:17.652579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.652580 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 14:18:17.652694 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.652591 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 14:18:17.654525 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.654511 2562 policy_none.go:47] "Start" Jul 28 14:18:17.696240 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696011 2562 manager.go:337] "Starting Device Plugin manager" Jul 28 14:18:17.696451 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.696354 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 14:18:17.696451 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696370 2562 server.go:87] "Starting device plugin registration server" Jul 28 14:18:17.696767 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696750 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 14:18:17.696879 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696767 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 14:18:17.696930 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696872 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 14:18:17.696975 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696969 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 14:18:17.697027 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.696978 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 14:18:17.698296 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.698265 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 14:18:17.698394 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.698334 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:17.709534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.709505 2562 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2595 into the system container \"/system.slice\": " Jul 28 14:18:17.758437 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.758391 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 14:18:17.760706 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.759676 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 14:18:17.760706 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.759715 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 14:18:17.760706 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.759743 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 14:18:17.760706 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.759782 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 14:18:17.762554 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.762536 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 14:18:17.797809 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.797720 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.798804 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.798786 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.798895 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.798818 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.798895 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.798830 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.798895 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.798855 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.809017 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.808989 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.809085 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.809024 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-121.ec2.internal\": node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:17.854779 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.854747 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:17.860835 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.860806 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal"] Jul 28 14:18:17.860911 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.860885 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.861869 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.861853 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.861932 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.861883 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.861932 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.861894 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.864260 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.864246 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.864384 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.864369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.864443 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.864402 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.865107 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.865088 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.865195 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.865117 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.865195 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.865129 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.866368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.866339 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.866459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.866375 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.866459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.866387 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.868673 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.868658 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.868735 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.868688 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 14:18:17.869536 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.869518 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientMemory" Jul 28 14:18:17.869635 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.869544 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 14:18:17.869635 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.869554 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeHasSufficientPID" Jul 28 14:18:17.887017 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.886989 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.892064 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.892041 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-121.ec2.internal\" not found" node="ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.926046 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.926015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.926233 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.926062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.926233 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:17.926083 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c5d5894bca76db08e7d990578b15cf0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"0c5d5894bca76db08e7d990578b15cf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 28 14:18:17.955680 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:17.955641 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.026983 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.026945 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.027096 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.026987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.027096 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.027054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c5d5894bca76db08e7d990578b15cf0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"0c5d5894bca76db08e7d990578b15cf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.027096 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.027083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.027225 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.027123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c3ed888f968446d5f752a5eb72daf11a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal\" (UID: \"c3ed888f968446d5f752a5eb72daf11a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.027225 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.027126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c5d5894bca76db08e7d990578b15cf0-config\") pod \"kube-apiserver-proxy-ip-10-0-131-121.ec2.internal\" (UID: \"0c5d5894bca76db08e7d990578b15cf0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.056108 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:18.056032 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.156959 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:18.156919 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.190139 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.190110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.196359 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.196337 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.257445 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:18.257390 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.357944 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:18.357860 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.458517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:18.458474 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.556065 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.556021 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 14:18:18.556597 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.556225 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 14:18:18.559194 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:18.559164 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-121.ec2.internal\" not found" Jul 28 14:18:18.626709 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.626451 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 14:18:18.654219 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.654173 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 14:13:18 +0000 UTC" deadline="2028-01-21 02:53:59.881626865 +0000 UTC" Jul 28 14:18:18.654425 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.654232 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12996h35m41.227417247s" Jul 28 14:18:18.723900 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.723867 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.729743 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.729719 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 14:18:18.738446 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.738418 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 14:18:18.739381 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.739362 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" Jul 28 14:18:18.742683 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:18.742645 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5d5894bca76db08e7d990578b15cf0.slice/crio-1254d15fe9e103106af5dfbd13fe08a568e725d365a0ddf0eac8f25fe3337d5c WatchSource:0}: Error finding container 1254d15fe9e103106af5dfbd13fe08a568e725d365a0ddf0eac8f25fe3337d5c: Status 404 returned error can't find the container with id 1254d15fe9e103106af5dfbd13fe08a568e725d365a0ddf0eac8f25fe3337d5c Jul 28 14:18:18.748956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.748939 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 14:18:18.754313 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.754294 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 14:18:18.760107 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:18.760077 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ed888f968446d5f752a5eb72daf11a.slice/crio-c3489997d1d80fff3ac6cf3a82a8a5234d3a8b5e073c82b02cb5a3a83ab010e9 WatchSource:0}: Error finding container c3489997d1d80fff3ac6cf3a82a8a5234d3a8b5e073c82b02cb5a3a83ab010e9: Status 404 returned error can't find the container with id c3489997d1d80fff3ac6cf3a82a8a5234d3a8b5e073c82b02cb5a3a83ab010e9 Jul 28 14:18:18.763004 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.762949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"c3ed888f968446d5f752a5eb72daf11a","Type":"ContainerStarted","Data":"c3489997d1d80fff3ac6cf3a82a8a5234d3a8b5e073c82b02cb5a3a83ab010e9"} Jul 28 14:18:18.764050 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:18.764022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" event={"ID":"0c5d5894bca76db08e7d990578b15cf0","Type":"ContainerStarted","Data":"1254d15fe9e103106af5dfbd13fe08a568e725d365a0ddf0eac8f25fe3337d5c"} Jul 28 14:18:19.102595 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.102503 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 14:18:19.599742 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.599652 2562 apiserver.go:52] "Watching apiserver" Jul 28 14:18:19.606880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.606840 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 14:18:19.607477 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.607450 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-fpqnz","openshift-ovn-kubernetes/ovnkube-node-brkdl","kube-system/global-pull-secret-syncer-fgmfx","kube-system/konnectivity-agent-bb4bs","openshift-image-registry/node-ca-ttdk2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal","openshift-multus/multus-869vc","openshift-multus/multus-additional-cni-plugins-8vgkc","openshift-multus/network-metrics-daemon-8sjnc","openshift-network-operator/iptables-alerter-tvnpl","kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k","openshift-cluster-node-tuning-operator/tuned-rh99m","openshift-dns/node-resolver-bs7j4"] Jul 28 14:18:19.610140 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.610116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.613287 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.612758 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:19.613287 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.612833 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:19.613478 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.613298 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bpmqf\"" Jul 28 14:18:19.613565 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.613539 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 14:18:19.613632 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.613578 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.614024 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.614006 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.615231 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.615213 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-869vc" Jul 28 14:18:19.616795 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.616774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.617432 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.617413 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.618714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.618693 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 14:18:19.618872 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.618855 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mqrkn\"" Jul 28 14:18:19.619006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.618985 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.619075 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.619033 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 14:18:19.619457 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.619436 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 14:18:19.619528 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.619516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.619848 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.619585 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:19.620533 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.620517 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hjvwv\"" Jul 28 14:18:19.620635 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.620616 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 14:18:19.622824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.622804 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.624028 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.624009 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 14:18:19.626904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.626554 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.626904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.626565 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.626904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.626605 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 14:18:19.626904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.626843 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 14:18:19.627531 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.627514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hkw49\"" Jul 28 14:18:19.627716 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.627517 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 14:18:19.628100 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.628080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.628896 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.628802 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 14:18:19.630558 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.630537 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 14:18:19.630655 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.630622 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 14:18:19.630788 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.630767 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5rndf\"" Jul 28 14:18:19.632902 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.632876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:19.633019 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.632990 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:19.634966 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.634940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-conf-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.635116 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.634985 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a38c15d-bd62-4e10-8c87-120662bd95f7-dbus\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.635116 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-systemd\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.635116 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxcc\" (UniqueName: \"kubernetes.io/projected/e1b33829-f61b-421b-9749-3abf813f9766-kube-api-access-ljxcc\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.635116 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-k8s-cni-cncf-io\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.635394 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-cni-bin\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.635394 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovn-node-metrics-cert\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.635394 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4f710d1-1eaa-45e8-8853-4bdc020989a0-serviceca\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.635394 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.635672 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-cni-multus\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.635672 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a38c15d-bd62-4e10-8c87-120662bd95f7-kubelet-config\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.635824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4f710d1-1eaa-45e8-8853-4bdc020989a0-host\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.635824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635715 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f2bf9-a419-4e40-a236-ca08eabe03bb-konnectivity-ca\") pod \"konnectivity-agent-bb4bs\" (UID: \"dc3f2bf9-a419-4e40-a236-ca08eabe03bb\") " pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.635824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-kubelet\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.635824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-ovn\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.635824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-cni-netd\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-netns\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-etc-kubernetes\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.635992 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkq8\" (UniqueName: \"kubernetes.io/projected/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-kube-api-access-5nkq8\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-cni-binary-copy\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-cnibin\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-etc-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovnkube-config\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636111 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-slash\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-log-socket\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-env-overrides\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6qhk\" (UniqueName: \"kubernetes.io/projected/a4f710d1-1eaa-45e8-8853-4bdc020989a0-kube-api-access-s6qhk\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-kubelet\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-daemon-config\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-system-cni-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-socket-dir-parent\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5cg\" (UniqueName: \"kubernetes.io/projected/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-kube-api-access-gs5cg\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-run-netns\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636530 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-cni-binary-copy\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-multus-certs\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.636622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovnkube-script-lib\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-hostroot\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-var-lib-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-node-log\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-system-cni-dir\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-os-release\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636842 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-os-release\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-cni-bin\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.636942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-cnibin\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637019 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637046 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-cni-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc3f2bf9-a419-4e40-a236-ca08eabe03bb-agent-certs\") pod \"konnectivity-agent-bb4bs\" (UID: \"dc3f2bf9-a419-4e40-a236-ca08eabe03bb\") " pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.637564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-systemd-units\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.638521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637606 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.638521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637728 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.638521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h5frk\"" Jul 28 14:18:19.638521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.637917 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 14:18:19.638521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.638457 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.640464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.640436 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.640719 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.640693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ck7g4\"" Jul 28 14:18:19.640805 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.640777 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 14:18:19.641093 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.641075 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.641500 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.641481 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 14:18:19.641837 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.641815 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.643924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.643900 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.644084 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.644063 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.646459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.646303 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.646550 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.646472 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 14:18:19.647168 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.646842 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 14:18:19.647168 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.646947 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sl79r\"" Jul 28 14:18:19.647168 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.647101 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k86n8\"" Jul 28 14:18:19.660473 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.660449 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 14:18:19.669904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.669848 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7jlkk" Jul 28 14:18:19.679500 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.679472 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7jlkk" Jul 28 14:18:19.725772 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.725741 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 14:18:19.737389 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.737389 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-hostroot\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-var-lib-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-node-log\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-system-cni-dir\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-os-release\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-hostroot\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737496 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-var-lib-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-host\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-node-log\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-os-release\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737567 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-system-cni-dir\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737591 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-os-release\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.737627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-cni-bin\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-cnibin\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737685 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-cni-bin\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-os-release\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-cnibin\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737758 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysctl-conf\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-cni-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc3f2bf9-a419-4e40-a236-ca08eabe03bb-agent-certs\") pod \"konnectivity-agent-bb4bs\" (UID: \"dc3f2bf9-a419-4e40-a236-ca08eabe03bb\") " pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-systemd-units\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-device-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-conf-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a38c15d-bd62-4e10-8c87-120662bd95f7-dbus\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737910 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-systemd-units\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-systemd\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737953 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-conf-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-systemd\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.738334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.737984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxcc\" (UniqueName: \"kubernetes.io/projected/e1b33829-f61b-421b-9749-3abf813f9766-kube-api-access-ljxcc\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738017 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-cni-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738021 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a38c15d-bd62-4e10-8c87-120662bd95f7-dbus\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-lib-modules\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-systemd\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-tmp\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738192 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738200 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktklr\" (UniqueName: \"kubernetes.io/projected/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-kube-api-access-ktklr\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738300 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-k8s-cni-cncf-io\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-cni-bin\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovn-node-metrics-cert\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4f710d1-1eaa-45e8-8853-4bdc020989a0-serviceca\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-sys-fs\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-cni-bin\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738417 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f842320b-70c6-466f-9e1c-fcf38d578666-hosts-file\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-cni-multus\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a38c15d-bd62-4e10-8c87-120662bd95f7-kubelet-config\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.739149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4f710d1-1eaa-45e8-8853-4bdc020989a0-host\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26aee6ea-17fa-4de5-8875-10a9003aba6e-iptables-alerter-script\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-sys\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f2bf9-a419-4e40-a236-ca08eabe03bb-konnectivity-ca\") pod \"konnectivity-agent-bb4bs\" (UID: \"dc3f2bf9-a419-4e40-a236-ca08eabe03bb\") " pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-kubelet\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-ovn\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-cni-netd\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738738 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-modprobe-d\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738764 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-k8s-cni-cncf-io\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-cni-multus\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4f710d1-1eaa-45e8-8853-4bdc020989a0-host\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.739943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738807 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4f710d1-1eaa-45e8-8853-4bdc020989a0-serviceca\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738836 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-kubelet\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a38c15d-bd62-4e10-8c87-120662bd95f7-kubelet-config\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-cni-netd\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-netns\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-run-ovn\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-etc-kubernetes\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738997 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-etc-kubernetes\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.738996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkq8\" (UniqueName: \"kubernetes.io/projected/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-kube-api-access-5nkq8\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739065 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-netns\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-cni-binary-copy\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739139 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b33829-f61b-421b-9749-3abf813f9766-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-run\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f842320b-70c6-466f-9e1c-fcf38d578666-tmp-dir\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.740682 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgc2k\" (UniqueName: \"kubernetes.io/projected/a8fe04c7-9567-4cbe-9128-54c5456ac568-kube-api-access-cgc2k\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-cnibin\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-etc-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc3f2bf9-a419-4e40-a236-ca08eabe03bb-konnectivity-ca\") pod \"konnectivity-agent-bb4bs\" (UID: \"dc3f2bf9-a419-4e40-a236-ca08eabe03bb\") " pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovnkube-config\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739389 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgb6z\" (UniqueName: \"kubernetes.io/projected/f0fded26-1385-4fb1-93df-dea08d2b474c-kube-api-access-dgb6z\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-socket-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-cnibin\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-kubernetes\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-etc-openvswitch\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-slash\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-log-socket\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.739620 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-log-socket\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-slash\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-env-overrides\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.741459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6qhk\" (UniqueName: \"kubernetes.io/projected/a4f710d1-1eaa-45e8-8853-4bdc020989a0-kube-api-access-s6qhk\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.739748 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:20.239713216 +0000 UTC m=+3.027353804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b33829-f61b-421b-9749-3abf813f9766-cni-binary-copy\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovnkube-config\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2hn\" (UniqueName: \"kubernetes.io/projected/26aee6ea-17fa-4de5-8875-10a9003aba6e-kube-api-access-5r2hn\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-registration-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-kubelet\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-daemon-config\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.739979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-var-lib-kubelet\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26aee6ea-17fa-4de5-8875-10a9003aba6e-host-slash\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysconfig\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740490 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-env-overrides\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysctl-d\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-var-lib-kubelet\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740613 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-tuned\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.742259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-system-cni-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-socket-dir-parent\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5cg\" (UniqueName: \"kubernetes.io/projected/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-kube-api-access-gs5cg\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-run-netns\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-cni-binary-copy\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-multus-certs\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovnkube-script-lib\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.740938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2ths\" (UniqueName: \"kubernetes.io/projected/f842320b-70c6-466f-9e1c-fcf38d578666-kube-api-access-w2ths\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.741147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-daemon-config\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.741155 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-system-cni-dir\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.741202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-multus-socket-dir-parent\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.741296 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-host-run-netns\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.741306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-host-run-multus-certs\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.742087 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-cni-binary-copy\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.742292 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovnkube-script-lib\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.742981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.742707 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-ovn-node-metrics-cert\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.745663 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.745637 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc3f2bf9-a419-4e40-a236-ca08eabe03bb-agent-certs\") pod \"konnectivity-agent-bb4bs\" (UID: \"dc3f2bf9-a419-4e40-a236-ca08eabe03bb\") " pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:19.747555 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.747508 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:19.747676 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.747561 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:19.747676 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.747578 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:19.747931 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.747912 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:18:20.247667987 +0000 UTC m=+3.035308595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:19.748625 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.748604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxcc\" (UniqueName: \"kubernetes.io/projected/e1b33829-f61b-421b-9749-3abf813f9766-kube-api-access-ljxcc\") pod \"multus-additional-cni-plugins-8vgkc\" (UID: \"e1b33829-f61b-421b-9749-3abf813f9766\") " pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:19.748814 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.748760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkq8\" (UniqueName: \"kubernetes.io/projected/17ca5711-b9a9-4b33-b8e8-cecf9dc090f4-kube-api-access-5nkq8\") pod \"ovnkube-node-brkdl\" (UID: \"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:19.748966 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.748941 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6qhk\" (UniqueName: \"kubernetes.io/projected/a4f710d1-1eaa-45e8-8853-4bdc020989a0-kube-api-access-s6qhk\") pod \"node-ca-ttdk2\" (UID: \"a4f710d1-1eaa-45e8-8853-4bdc020989a0\") " pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:19.749103 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.749086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5cg\" (UniqueName: \"kubernetes.io/projected/cacbe418-1ee3-4d2f-b007-00f6f02b7c9b-kube-api-access-gs5cg\") pod \"multus-869vc\" (UID: \"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b\") " pod="openshift-multus/multus-869vc" Jul 28 14:18:19.841765 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.841728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-socket-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.841765 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.841767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-kubernetes\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.841906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2hn\" (UniqueName: \"kubernetes.io/projected/26aee6ea-17fa-4de5-8875-10a9003aba6e-kube-api-access-5r2hn\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.842019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.841949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-registration-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.842019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.841954 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-socket-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.842019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.841981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26aee6ea-17fa-4de5-8875-10a9003aba6e-host-slash\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.842019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-kubernetes\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysconfig\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842025 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-registration-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysctl-d\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/26aee6ea-17fa-4de5-8875-10a9003aba6e-host-slash\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-var-lib-kubelet\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-tuned\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysconfig\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842131 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-var-lib-kubelet\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysctl-d\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2ths\" (UniqueName: \"kubernetes.io/projected/f842320b-70c6-466f-9e1c-fcf38d578666-kube-api-access-w2ths\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.842262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-host\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysctl-conf\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-host\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-device-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-systemd\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.842326 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-lib-modules\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842379 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-device-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-tmp\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-sysctl-conf\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:19.842434 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:18:20.34240962 +0000 UTC m=+3.130050224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842437 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-systemd\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktklr\" (UniqueName: \"kubernetes.io/projected/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-kube-api-access-ktklr\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-sys-fs\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f842320b-70c6-466f-9e1c-fcf38d578666-hosts-file\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-lib-modules\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26aee6ea-17fa-4de5-8875-10a9003aba6e-iptables-alerter-script\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.842776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-sys-fs\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f842320b-70c6-466f-9e1c-fcf38d578666-hosts-file\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-sys\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-sys\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-modprobe-d\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842700 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-run\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f842320b-70c6-466f-9e1c-fcf38d578666-tmp-dir\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8fe04c7-9567-4cbe-9128-54c5456ac568-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgc2k\" (UniqueName: \"kubernetes.io/projected/a8fe04c7-9567-4cbe-9128-54c5456ac568-kube-api-access-cgc2k\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-modprobe-d\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgb6z\" (UniqueName: \"kubernetes.io/projected/f0fded26-1385-4fb1-93df-dea08d2b474c-kube-api-access-dgb6z\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.842797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-run\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.843617 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.843076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f842320b-70c6-466f-9e1c-fcf38d578666-tmp-dir\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.844173 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.843709 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/26aee6ea-17fa-4de5-8875-10a9003aba6e-iptables-alerter-script\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.844788 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.844758 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-tmp\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.844924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.844896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-etc-tuned\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.852688 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.852620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2hn\" (UniqueName: \"kubernetes.io/projected/26aee6ea-17fa-4de5-8875-10a9003aba6e-kube-api-access-5r2hn\") pod \"iptables-alerter-tvnpl\" (UID: \"26aee6ea-17fa-4de5-8875-10a9003aba6e\") " pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:19.853620 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.853595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2ths\" (UniqueName: \"kubernetes.io/projected/f842320b-70c6-466f-9e1c-fcf38d578666-kube-api-access-w2ths\") pod \"node-resolver-bs7j4\" (UID: \"f842320b-70c6-466f-9e1c-fcf38d578666\") " pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:19.856971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.856937 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktklr\" (UniqueName: \"kubernetes.io/projected/b36a6a4b-7e1a-4feb-9e62-e174c6255f50-kube-api-access-ktklr\") pod \"tuned-rh99m\" (UID: \"b36a6a4b-7e1a-4feb-9e62-e174c6255f50\") " pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:19.857447 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.857422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgb6z\" (UniqueName: \"kubernetes.io/projected/f0fded26-1385-4fb1-93df-dea08d2b474c-kube-api-access-dgb6z\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:19.859389 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:19.859371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgc2k\" (UniqueName: \"kubernetes.io/projected/a8fe04c7-9567-4cbe-9128-54c5456ac568-kube-api-access-cgc2k\") pod \"aws-ebs-csi-driver-node-wrs2k\" (UID: \"a8fe04c7-9567-4cbe-9128-54c5456ac568\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:21.045722 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.045685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ttdk2" Jul 28 14:18:21.055316 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.055288 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-869vc" Jul 28 14:18:21.066286 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.066257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:21.073265 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.073239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:21.081064 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.081041 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" Jul 28 14:18:21.090736 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.090705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tvnpl" Jul 28 14:18:21.098649 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.098619 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" Jul 28 14:18:21.105637 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.105612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rh99m" Jul 28 14:18:21.112566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.112542 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bs7j4" Jul 28 14:18:21.369140 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.369046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:21.369306 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.369214 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:21.369306 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.369302 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:22.369279407 +0000 UTC m=+4.034521225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:21.469954 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.469910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:21.469954 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.469963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:21.470227 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.470059 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:21.470227 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.470081 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:21.470227 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.470093 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:21.470227 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.470133 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:21.470227 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.470164 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:18:22.470145476 +0000 UTC m=+4.135387294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:21.470227 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:21.470201 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:18:22.470183958 +0000 UTC m=+4.135425762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:21.805080 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.804948 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 14:13:20 +0000 UTC" deadline="2028-04-10 06:49:46.160980024 +0000 UTC" Jul 28 14:18:21.805080 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.805015 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14920h31m24.355969802s" Jul 28 14:18:21.858737 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.858646 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fe04c7_9567_4cbe_9128_54c5456ac568.slice/crio-518a9c0b67dde8750741e808a2357388219a2a4e670288e42b9796c877b1ed8e WatchSource:0}: Error finding container 518a9c0b67dde8750741e808a2357388219a2a4e670288e42b9796c877b1ed8e: Status 404 returned error can't find the container with id 518a9c0b67dde8750741e808a2357388219a2a4e670288e42b9796c877b1ed8e Jul 28 14:18:21.881146 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.880998 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26aee6ea_17fa_4de5_8875_10a9003aba6e.slice/crio-0d0c25c7832ba65b445bb1fc1bc98afe298403dd75f7b1150a0ee151447ba4b2 WatchSource:0}: Error finding container 0d0c25c7832ba65b445bb1fc1bc98afe298403dd75f7b1150a0ee151447ba4b2: Status 404 returned error can't find the container with id 0d0c25c7832ba65b445bb1fc1bc98afe298403dd75f7b1150a0ee151447ba4b2 Jul 28 14:18:21.883613 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.883585 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f710d1_1eaa_45e8_8853_4bdc020989a0.slice/crio-1381fc4fa684c635551865912701b98b2d5dd4e3b2f9198691e3f9d72768fe88 WatchSource:0}: Error finding container 1381fc4fa684c635551865912701b98b2d5dd4e3b2f9198691e3f9d72768fe88: Status 404 returned error can't find the container with id 1381fc4fa684c635551865912701b98b2d5dd4e3b2f9198691e3f9d72768fe88 Jul 28 14:18:21.884383 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.884314 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3f2bf9_a419_4e40_a236_ca08eabe03bb.slice/crio-449a4fddad57216a1b7e7ee306c546f7ebe391b4651eaa686212210278a0de93 WatchSource:0}: Error finding container 449a4fddad57216a1b7e7ee306c546f7ebe391b4651eaa686212210278a0de93: Status 404 returned error can't find the container with id 449a4fddad57216a1b7e7ee306c546f7ebe391b4651eaa686212210278a0de93 Jul 28 14:18:21.885643 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.885620 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36a6a4b_7e1a_4feb_9e62_e174c6255f50.slice/crio-d60c16e7704a0f6068262f750ae9656f041173f605223dd5aa0fede6d9d0c2fd WatchSource:0}: Error finding container d60c16e7704a0f6068262f750ae9656f041173f605223dd5aa0fede6d9d0c2fd: Status 404 returned error can't find the container with id d60c16e7704a0f6068262f750ae9656f041173f605223dd5aa0fede6d9d0c2fd Jul 28 14:18:21.887055 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.887023 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ca5711_b9a9_4b33_b8e8_cecf9dc090f4.slice/crio-7c9fd6fa669e5052e26c3a1826f51ad19284ae76e963566a0c3075019c1161ea WatchSource:0}: Error finding container 7c9fd6fa669e5052e26c3a1826f51ad19284ae76e963566a0c3075019c1161ea: Status 404 returned error can't find the container with id 7c9fd6fa669e5052e26c3a1826f51ad19284ae76e963566a0c3075019c1161ea Jul 28 14:18:21.887781 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:21.887760 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf842320b_70c6_466f_9e1c_fcf38d578666.slice/crio-d5f379f8fa6d9492d5bbf053c19704a42e62e2bf984c7aa9888179441dcc2124 WatchSource:0}: Error finding container d5f379f8fa6d9492d5bbf053c19704a42e62e2bf984c7aa9888179441dcc2124: Status 404 returned error can't find the container with id d5f379f8fa6d9492d5bbf053c19704a42e62e2bf984c7aa9888179441dcc2124 Jul 28 14:18:21.893247 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.893095 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"7c9fd6fa669e5052e26c3a1826f51ad19284ae76e963566a0c3075019c1161ea"} Jul 28 14:18:21.895232 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.894935 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rh99m" event={"ID":"b36a6a4b-7e1a-4feb-9e62-e174c6255f50","Type":"ContainerStarted","Data":"d60c16e7704a0f6068262f750ae9656f041173f605223dd5aa0fede6d9d0c2fd"} Jul 28 14:18:21.897759 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.897660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb4bs" event={"ID":"dc3f2bf9-a419-4e40-a236-ca08eabe03bb","Type":"ContainerStarted","Data":"449a4fddad57216a1b7e7ee306c546f7ebe391b4651eaa686212210278a0de93"} Jul 28 14:18:21.900189 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.900163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ttdk2" event={"ID":"a4f710d1-1eaa-45e8-8853-4bdc020989a0","Type":"ContainerStarted","Data":"1381fc4fa684c635551865912701b98b2d5dd4e3b2f9198691e3f9d72768fe88"} Jul 28 14:18:21.901726 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.901701 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvnpl" event={"ID":"26aee6ea-17fa-4de5-8875-10a9003aba6e","Type":"ContainerStarted","Data":"0d0c25c7832ba65b445bb1fc1bc98afe298403dd75f7b1150a0ee151447ba4b2"} Jul 28 14:18:21.902895 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:21.902872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" event={"ID":"a8fe04c7-9567-4cbe-9128-54c5456ac568","Type":"ContainerStarted","Data":"518a9c0b67dde8750741e808a2357388219a2a4e670288e42b9796c877b1ed8e"} Jul 28 14:18:22.377475 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.377366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:22.377648 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.377544 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:22.377648 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.377623 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:24.377601504 +0000 UTC m=+6.042843305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:22.478331 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.478291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:22.478517 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.478345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:22.478517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.478471 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:22.478517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.478484 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:22.478517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.478512 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:22.478745 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.478528 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:22.478745 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.478536 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:18:24.478515845 +0000 UTC m=+6.143757649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:22.478745 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.478581 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:18:24.478564254 +0000 UTC m=+6.143806080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:22.805738 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.805594 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 14:13:20 +0000 UTC" deadline="2028-02-16 04:43:59.205778164 +0000 UTC" Jul 28 14:18:22.805738 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.805639 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13622h25m36.400144101s" Jul 28 14:18:22.885530 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.885145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:22.885530 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.885296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:22.885530 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.885298 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:22.885530 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.885359 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:22.885530 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.885446 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:22.885942 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:22.885550 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:22.912706 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.912600 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" event={"ID":"0c5d5894bca76db08e7d990578b15cf0","Type":"ContainerStarted","Data":"856d4c139af35b6d01690dd40b72ee3d66562a239a774cc4256ba99e7c5d7de8"} Jul 28 14:18:22.917197 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.917118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-869vc" event={"ID":"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b","Type":"ContainerStarted","Data":"a5339c42b135915b90fc0bc7fcf1728202f49fc6ce43ff8750772a851764fdf5"} Jul 28 14:18:22.919518 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.919479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerStarted","Data":"857608778c4a42ec8b3648118d32eed5db5f037cda70eaacc7d037be52adc2f9"} Jul 28 14:18:22.921423 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.921392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bs7j4" event={"ID":"f842320b-70c6-466f-9e1c-fcf38d578666","Type":"ContainerStarted","Data":"d5f379f8fa6d9492d5bbf053c19704a42e62e2bf984c7aa9888179441dcc2124"} Jul 28 14:18:22.923633 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.923598 2562 generic.go:356] "Generic (PLEG): container finished" podID="c3ed888f968446d5f752a5eb72daf11a" containerID="36513e3368860048f3b64f87efa02e0d269e1297ce53b0a8cc1d2e59400f7239" exitCode=0 Jul 28 14:18:22.923778 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.923649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"c3ed888f968446d5f752a5eb72daf11a","Type":"ContainerDied","Data":"36513e3368860048f3b64f87efa02e0d269e1297ce53b0a8cc1d2e59400f7239"} Jul 28 14:18:22.954122 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:22.954051 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-121.ec2.internal" podStartSLOduration=3.954024863 podStartE2EDuration="3.954024863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:22.934012581 +0000 UTC m=+4.599254407" watchObservedRunningTime="2026-07-28 14:18:22.954024863 +0000 UTC m=+4.619266687" Jul 28 14:18:23.929349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:23.929306 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" event={"ID":"c3ed888f968446d5f752a5eb72daf11a","Type":"ContainerStarted","Data":"8ed375f6575fde6cb4a18870080844e96a20c69d64f5940404bdfdb2341d3a00"} Jul 28 14:18:23.954576 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:23.954509 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-121.ec2.internal" podStartSLOduration=4.954485911 podStartE2EDuration="4.954485911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:23.954091362 +0000 UTC m=+5.619333179" watchObservedRunningTime="2026-07-28 14:18:23.954485911 +0000 UTC m=+5.619727735" Jul 28 14:18:24.397276 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:24.397186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:24.397466 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.397332 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:24.397466 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.397431 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:28.397409009 +0000 UTC m=+10.062650813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:24.497870 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:24.497825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:24.498071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:24.497882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:24.498071 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.498002 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:24.498071 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.498032 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:24.498071 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.498057 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:24.498071 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.498064 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:18:28.498044696 +0000 UTC m=+10.163286518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:24.498071 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.498070 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:24.498397 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.498121 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:18:28.498104976 +0000 UTC m=+10.163346794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:24.883914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:24.883269 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:24.883914 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.883430 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:24.883914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:24.883727 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:24.883914 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.883834 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:24.884295 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:24.883991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:24.884295 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:24.884088 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:26.883649 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:26.883612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:26.883649 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:26.883653 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:26.884211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:26.883612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:26.884211 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:26.883782 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:26.884211 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:26.883830 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:26.884211 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:26.883891 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:28.428624 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:28.428508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:28.429125 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.428694 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:28.429125 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.428774 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:36.428751127 +0000 UTC m=+18.093992931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:28.529559 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:28.529515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:28.529731 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:28.529568 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:28.529780 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.529730 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:28.529843 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.529793 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:18:36.52977408 +0000 UTC m=+18.195015886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:28.530361 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.530236 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:28.530361 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.530264 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:28.530361 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.530278 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:28.530361 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.530328 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:18:36.530310517 +0000 UTC m=+18.195552336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:28.884132 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:28.883925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:28.884132 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:28.884026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:28.884132 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.884052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:28.884427 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.884160 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:28.884427 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:28.884227 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:28.884427 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:28.884314 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:30.883555 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.883291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:30.884024 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.883291 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:30.884024 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:30.883617 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:30.884024 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.883406 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:30.884024 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:30.883657 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:30.884024 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:30.883814 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:30.950085 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.950047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ttdk2" event={"ID":"a4f710d1-1eaa-45e8-8853-4bdc020989a0","Type":"ContainerStarted","Data":"7b27dac49a4a1b08053b14e3b44fdd31545b97160fc2b532f438968c7e76cacc"} Jul 28 14:18:30.952642 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.952605 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" event={"ID":"a8fe04c7-9567-4cbe-9128-54c5456ac568","Type":"ContainerStarted","Data":"f5375d49534c5978943ce4509bc0ee74bf44f7471271f0ff64dad2b886caba4b"} Jul 28 14:18:30.954792 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.954749 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1b33829-f61b-421b-9749-3abf813f9766" containerID="9a99c8d2ae882fcb04aa1ee864af51452bc72c45f9f79882687b6dfab131b7e5" exitCode=0 Jul 28 14:18:30.954939 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.954857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerDied","Data":"9a99c8d2ae882fcb04aa1ee864af51452bc72c45f9f79882687b6dfab131b7e5"} Jul 28 14:18:30.958102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.958068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bs7j4" event={"ID":"f842320b-70c6-466f-9e1c-fcf38d578666","Type":"ContainerStarted","Data":"f45bdf0d5edebb6baaee2e8fb35c8598f7ec6788e507acc6172c78320a895713"} Jul 28 14:18:30.960275 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.960246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rh99m" event={"ID":"b36a6a4b-7e1a-4feb-9e62-e174c6255f50","Type":"ContainerStarted","Data":"ce0fdb13d8589c8c32db3853aec5a04725ce852aa5822d625742906ff4bef89c"} Jul 28 14:18:30.963347 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.963313 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb4bs" event={"ID":"dc3f2bf9-a419-4e40-a236-ca08eabe03bb","Type":"ContainerStarted","Data":"0ba228f5047c76e28e06cf00dc6f8aa95060ccb7e29286c71b9014fe1cb7982f"} Jul 28 14:18:30.972117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:30.971329 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ttdk2" podStartSLOduration=4.470720916 podStartE2EDuration="12.97130808s" totalImagesPullingTime="8.500587164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:30.970986762 +0000 UTC m=+12.636228587" watchObservedRunningTime="2026-07-28 14:18:30.97130808 +0000 UTC m=+12.636549904" Jul 28 14:18:31.031564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:31.031493 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rh99m" podStartSLOduration=3.619545941 podStartE2EDuration="12.031466955s" totalImagesPullingTime="8.411921014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:31.030456397 +0000 UTC m=+12.695698221" watchObservedRunningTime="2026-07-28 14:18:31.031466955 +0000 UTC m=+12.696708780" Jul 28 14:18:31.080618 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:31.080552 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bs7j4" podStartSLOduration=3.671771599 podStartE2EDuration="12.080529563s" totalImagesPullingTime="8.408757964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:31.054114452 +0000 UTC m=+12.719356276" watchObservedRunningTime="2026-07-28 14:18:31.080529563 +0000 UTC m=+12.745771389" Jul 28 14:18:31.080979 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:31.080938 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bb4bs" podStartSLOduration=3.670532128 podStartE2EDuration="12.080925593s" totalImagesPullingTime="8.410393465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:31.080042089 +0000 UTC m=+12.745283935" watchObservedRunningTime="2026-07-28 14:18:31.080925593 +0000 UTC m=+12.746167416" Jul 28 14:18:31.967612 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:31.967569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tvnpl" event={"ID":"26aee6ea-17fa-4de5-8875-10a9003aba6e","Type":"ContainerStarted","Data":"4dee02b3a2cbad7365c46d0c694e1de13989217c9efd7175fe7af561a8103df0"} Jul 28 14:18:31.984870 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:31.984778 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tvnpl" podStartSLOduration=4.568120932 podStartE2EDuration="12.984752861s" totalImagesPullingTime="8.416631929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:31.984275297 +0000 UTC m=+13.649517121" watchObservedRunningTime="2026-07-28 14:18:31.984752861 +0000 UTC m=+13.649994687" Jul 28 14:18:32.883229 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:32.883192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:32.883406 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:32.883353 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:32.883778 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:32.883757 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:32.883894 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:32.883865 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:32.884023 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:32.884004 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:32.884122 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:32.884103 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:34.400691 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.400644 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:34.401961 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.401890 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:34.882869 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.882787 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:34.883016 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.882787 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:34.883016 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:34.882922 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:34.883016 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.882787 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:34.883016 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:34.882979 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:34.883203 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:34.883077 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:34.925234 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.925195 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:34.925933 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:34.925915 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bb4bs" Jul 28 14:18:36.497597 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:36.497551 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:36.498281 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.497733 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:36.498281 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.497815 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:52.497791926 +0000 UTC m=+34.163033728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:36.598856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:36.598793 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:36.598856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:36.598851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:36.599144 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.599027 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:36.599144 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.599059 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:36.599144 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.599093 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:36.599144 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.599110 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:36.599144 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.599118 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:18:52.599094676 +0000 UTC m=+34.264336497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:36.599372 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.599169 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:18:52.599149486 +0000 UTC m=+34.264391309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:36.883388 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:36.883296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:36.883562 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:36.883297 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:36.883562 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:36.883297 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:36.883562 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.883537 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:36.883772 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.883618 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:36.883772 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:36.883417 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:38.883359 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:38.883325 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:38.883867 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:38.883419 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:38.883867 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:38.883427 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:38.883867 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:38.883535 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:38.883867 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:38.883588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:38.883867 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:38.883641 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:40.882924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:40.882884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:40.883464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:40.883013 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:40.883464 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:40.883016 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:40.883464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:40.883029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:40.883464 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:40.883169 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:40.883464 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:40.883262 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:41.991119 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:41.990926 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"587992e906c6f81262815be14d77b208b63f055e40a46640013999be4a21a3fb"} Jul 28 14:18:41.991531 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:41.991205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"4388d8d86cf901b8c0b9fd447df64d9a0151e00fa9bb8e0a87624474f3843fdb"} Jul 28 14:18:41.992248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:41.992222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-869vc" event={"ID":"cacbe418-1ee3-4d2f-b007-00f6f02b7c9b","Type":"ContainerStarted","Data":"ee78805e42eea9bd17cd87310db83ef797b322afdfb5c1faa8286cae3809cc84"} Jul 28 14:18:42.016873 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.016826 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-869vc" podStartSLOduration=3.236060229 podStartE2EDuration="23.016811873s" totalImagesPullingTime="19.780751644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:42.015814967 +0000 UTC m=+23.681056791" watchObservedRunningTime="2026-07-28 14:18:42.016811873 +0000 UTC m=+23.682053692" Jul 28 14:18:42.716757 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.716734 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 14:18:42.830712 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.830618 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T14:18:42.716752749Z","UUID":"bc9dea16-dd6e-47d0-95ce-9ebbbe2ce17b","Handler":null,"Name":"","Endpoint":""} Jul 28 14:18:42.831978 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.831953 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 14:18:42.832064 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.831985 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 14:18:42.882921 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.882888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:42.883076 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.882887 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:42.883076 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:42.883013 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:42.883192 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.882888 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:42.883192 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:42.883112 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:42.883192 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:42.883176 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:42.997209 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997179 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:18:42.998026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997505 2562 generic.go:356] "Generic (PLEG): container finished" podID="17ca5711-b9a9-4b33-b8e8-cecf9dc090f4" containerID="587992e906c6f81262815be14d77b208b63f055e40a46640013999be4a21a3fb" exitCode=1 Jul 28 14:18:42.998026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerDied","Data":"587992e906c6f81262815be14d77b208b63f055e40a46640013999be4a21a3fb"} Jul 28 14:18:42.998026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997720 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"35164c042e7f0fb9fdff0b52ed5f0fca80a51c3f9e2014b2a27a9c03cd159b46"} Jul 28 14:18:42.998026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"6ad2a4f8630953d1b04feaa07e31ce73003c86e8295c9d1a0cafe46bcfc49601"} Jul 28 14:18:42.998026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"6326d47a3c49710ac151ea4410f4d326be2e6e5d528753bb8319a01467eb5efd"} Jul 28 14:18:42.998026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.997834 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"ab086751c6074d9d3b6b7378830a572bc396c3cc07fbf7aa9a986fff848caa45"} Jul 28 14:18:42.999322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:42.999299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" event={"ID":"a8fe04c7-9567-4cbe-9128-54c5456ac568","Type":"ContainerStarted","Data":"5c95d252f34e3e99bb08051fd39054472dddbef5a1baa5cfe518bb9aed71093c"} Jul 28 14:18:43.001028 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:43.000995 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1b33829-f61b-421b-9749-3abf813f9766" containerID="6d39b7a488657480ba66aee4fe86a00ad688842b2fc274e1a2a0ab3c22dd2674" exitCode=0 Jul 28 14:18:43.001125 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:43.001083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerDied","Data":"6d39b7a488657480ba66aee4fe86a00ad688842b2fc274e1a2a0ab3c22dd2674"} Jul 28 14:18:44.882517 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:44.882483 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:44.883033 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:44.882612 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:44.883033 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:44.882710 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:44.883033 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:44.882799 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:44.883033 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:44.882845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:44.883033 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:44.882908 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:45.008615 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:45.008434 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" event={"ID":"a8fe04c7-9567-4cbe-9128-54c5456ac568","Type":"ContainerStarted","Data":"01022e37d424f5f843d793e8e85fd816bc6836bb07e227693f62d36ebb97d166"} Jul 28 14:18:45.012159 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:45.012137 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:18:45.012531 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:45.012510 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"8222d38b0e70f808b59f439fd7d295c94b071367c1f9556e726e95f5d37757e8"} Jul 28 14:18:45.028362 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:45.028306 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrs2k" podStartSLOduration=4.446690091 podStartE2EDuration="26.028287739s" totalImagesPullingTime="21.581597648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:45.027564718 +0000 UTC m=+26.692806541" watchObservedRunningTime="2026-07-28 14:18:45.028287739 +0000 UTC m=+26.693529562" Jul 28 14:18:46.016657 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:46.016629 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1b33829-f61b-421b-9749-3abf813f9766" containerID="acd47efdbb79e090226a0bcee5317638bc2c85997294ba34f4d17573c2831f57" exitCode=0 Jul 28 14:18:46.017044 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:46.016714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerDied","Data":"acd47efdbb79e090226a0bcee5317638bc2c85997294ba34f4d17573c2831f57"} Jul 28 14:18:46.882877 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:46.882848 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:46.883083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:46.882845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:46.883083 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:46.882972 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:46.883083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:46.882983 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:46.883083 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:46.883057 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:46.883282 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:46.883183 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:47.021028 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:47.020933 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1b33829-f61b-421b-9749-3abf813f9766" containerID="d91c4d9786ef620023eeb7163dae51bca028b7194ce5dfc3d8c08b5b87ad35ed" exitCode=0 Jul 28 14:18:47.021028 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:47.020974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerDied","Data":"d91c4d9786ef620023eeb7163dae51bca028b7194ce5dfc3d8c08b5b87ad35ed"} Jul 28 14:18:48.030988 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.030757 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:18:48.031586 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.031370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"266741c04a7c084000b0f6a70a71320ae2d6aa727ddd284c5135e12f814cc0ef"} Jul 28 14:18:48.031654 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.031618 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:48.031728 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.031707 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:48.031781 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.031756 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:48.031856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.031836 2562 scope.go:117] "RemoveContainer" containerID="587992e906c6f81262815be14d77b208b63f055e40a46640013999be4a21a3fb" Jul 28 14:18:48.050963 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.050939 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:48.051345 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.051327 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:18:48.884472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.884435 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:48.884868 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.884484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:48.884868 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:48.884578 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:48.884868 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:48.884618 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:48.884868 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:48.884648 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:48.884868 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:48.884741 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:49.038710 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.038669 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:18:49.039235 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.039110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" event={"ID":"17ca5711-b9a9-4b33-b8e8-cecf9dc090f4","Type":"ContainerStarted","Data":"9501209b72a400d78be5edcfabb03b3f13ddca23d08e02fc3a2c5d2935af50cb"} Jul 28 14:18:49.083472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.083400 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" podStartSLOduration=10.297568895 podStartE2EDuration="30.083376238s" totalImagesPullingTime="19.785807343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:49.078054839 +0000 UTC m=+30.743296663" watchObservedRunningTime="2026-07-28 14:18:49.083376238 +0000 UTC m=+30.748618063" Jul 28 14:18:49.242655 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.242571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8sjnc"] Jul 28 14:18:49.242833 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.242776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:49.242930 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:49.242903 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:49.245130 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.245099 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fgmfx"] Jul 28 14:18:49.245251 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.245212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:49.245321 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:49.245301 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:49.248349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.248321 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fpqnz"] Jul 28 14:18:49.248513 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:49.248443 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:49.248573 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:49.248543 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:50.883123 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:50.883085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:50.883572 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:50.883085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:50.883572 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:50.883237 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:50.883572 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:50.883100 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:50.883572 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:50.883349 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:50.883572 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:50.883489 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:52.050087 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.050049 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1b33829-f61b-421b-9749-3abf813f9766" containerID="99dc4db4b8908b7dd4320d01ce95fad40990836ba0eea743ddc5f2a686c9afde" exitCode=0 Jul 28 14:18:52.050447 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.050086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerDied","Data":"99dc4db4b8908b7dd4320d01ce95fad40990836ba0eea743ddc5f2a686c9afde"} Jul 28 14:18:52.514706 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.514607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:52.514840 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.514741 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:52.514840 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.514793 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret podName:5a38c15d-bd62-4e10-8c87-120662bd95f7 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:24.51477923 +0000 UTC m=+66.180021031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret") pod "global-pull-secret-syncer-fgmfx" (UID: "5a38c15d-bd62-4e10-8c87-120662bd95f7") : object "kube-system"/"original-pull-secret" not registered Jul 28 14:18:52.615217 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.615181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:52.615217 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.615217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:52.615429 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.615327 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:52.615429 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.615342 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 14:18:52.615429 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.615360 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 14:18:52.615429 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.615371 2562 projected.go:196] Error preparing data for projected volume kube-api-access-6kgml for pod openshift-network-diagnostics/network-check-target-fpqnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:52.615429 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.615394 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:24.615374424 +0000 UTC m=+66.280616248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 14:18:52.615429 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.615413 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml podName:ecb6e983-1363-43cf-aa9c-0626a42d476b nodeName:}" failed. No retries permitted until 2026-07-28 14:19:24.615402231 +0000 UTC m=+66.280644032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6kgml" (UniqueName: "kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml") pod "network-check-target-fpqnz" (UID: "ecb6e983-1363-43cf-aa9c-0626a42d476b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 14:18:52.882812 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.882737 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:52.882812 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.882737 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:52.883025 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.882832 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:52.883025 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.882888 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:52.883025 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:52.882939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:52.883025 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:52.882997 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:53.056137 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:53.056096 2562 generic.go:356] "Generic (PLEG): container finished" podID="e1b33829-f61b-421b-9749-3abf813f9766" containerID="5980d344796ae574078e3aebbe60cb0e4a22b99a23eb81591d668957adb5e746" exitCode=0 Jul 28 14:18:53.056653 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:53.056154 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerDied","Data":"5980d344796ae574078e3aebbe60cb0e4a22b99a23eb81591d668957adb5e746"} Jul 28 14:18:54.061312 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:54.061283 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" event={"ID":"e1b33829-f61b-421b-9749-3abf813f9766","Type":"ContainerStarted","Data":"eb28407c4a549a7ea9d37dec544e30f9b6adbb9e08b48bccb079c3f44c7c5bd1"} Jul 28 14:18:54.085122 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:54.085077 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8vgkc" podStartSLOduration=7.850872368 podStartE2EDuration="35.085063786s" totalImagesPullingTime="27.234191418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:54.084484571 +0000 UTC m=+35.749726394" watchObservedRunningTime="2026-07-28 14:18:54.085063786 +0000 UTC m=+35.750305609" Jul 28 14:18:54.882989 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:54.882958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:54.883202 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:54.882958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:54.883202 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:54.883069 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8sjnc" podUID="f0fded26-1385-4fb1-93df-dea08d2b474c" Jul 28 14:18:54.883202 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:54.883152 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fgmfx" podUID="5a38c15d-bd62-4e10-8c87-120662bd95f7" Jul 28 14:18:54.883202 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:54.883193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:54.883333 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:54.883252 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fpqnz" podUID="ecb6e983-1363-43cf-aa9c-0626a42d476b" Jul 28 14:18:55.629510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.629483 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-121.ec2.internal" event="NodeReady" Jul 28 14:18:55.629926 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.629582 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 14:18:55.668121 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.668089 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cbdbcdff6-gvqcz"] Jul 28 14:18:55.703132 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.703107 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vt9w5"] Jul 28 14:18:55.703559 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.703538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.705975 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.705951 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lj7bw\"" Jul 28 14:18:55.706362 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.706343 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 14:18:55.706457 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.706349 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 14:18:55.706516 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.706455 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 14:18:55.712060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.712038 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 14:18:55.717011 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.716995 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cbdbcdff6-gvqcz"] Jul 28 14:18:55.717071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.717021 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7qlhl"] Jul 28 14:18:55.717107 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.717099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.720314 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.720295 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zfw4b\"" Jul 28 14:18:55.720414 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.720300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 14:18:55.720414 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.720335 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 14:18:55.720531 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.720442 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 14:18:55.739070 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.739051 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vt9w5"] Jul 28 14:18:55.739164 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.739082 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7qlhl"] Jul 28 14:18:55.739204 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.739164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.741420 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.741401 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 14:18:55.741517 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.741404 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6f28j\"" Jul 28 14:18:55.741517 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.741408 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 14:18:55.757500 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.757481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:18:55.840124 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-bound-sa-token\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840217 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840217 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f183322-bfbb-4763-b013-72e012cfa318-tmp-dir\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.840217 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzvr\" (UniqueName: \"kubernetes.io/projected/7cc40c6d-de39-4283-bb66-f6aa7bb99ac0-kube-api-access-5bzvr\") pod \"ingress-canary-vt9w5\" (UID: \"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0\") " pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.840326 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-trusted-ca\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840326 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-certificates\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840326 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-installation-pull-secrets\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840326 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6f183322-bfbb-4763-b013-72e012cfa318-config-volume\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.840484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkmt\" (UniqueName: \"kubernetes.io/projected/6f183322-bfbb-4763-b013-72e012cfa318-kube-api-access-7kkmt\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.840484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-image-registry-private-configuration\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cc40c6d-de39-4283-bb66-f6aa7bb99ac0-cert\") pod \"ingress-canary-vt9w5\" (UID: \"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0\") " pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.840484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.840484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840438 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvk9p\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-kube-api-access-pvk9p\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.840658 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.840498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/917ce53a-a02b-4fa9-9c8d-37861fdba354-ca-trust-extracted\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.858659 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.858637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:18:55.940856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.940803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/917ce53a-a02b-4fa9-9c8d-37861fdba354-ca-trust-extracted\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.940856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.940835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-bound-sa-token\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.940856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.940852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941016 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.940870 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f183322-bfbb-4763-b013-72e012cfa318-tmp-dir\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.941016 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.940888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzvr\" (UniqueName: \"kubernetes.io/projected/7cc40c6d-de39-4283-bb66-f6aa7bb99ac0-kube-api-access-5bzvr\") pod \"ingress-canary-vt9w5\" (UID: \"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0\") " pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.941016 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:55.940986 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:18:55.941016 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:55.941005 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:18:55.941201 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-trusted-ca\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941201 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:55.941129 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:56.441105049 +0000 UTC m=+38.106346859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:18:55.941201 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-certificates\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-installation-pull-secrets\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6f183322-bfbb-4763-b013-72e012cfa318-config-volume\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.941361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkmt\" (UniqueName: \"kubernetes.io/projected/6f183322-bfbb-4763-b013-72e012cfa318-kube-api-access-7kkmt\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.941361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/917ce53a-a02b-4fa9-9c8d-37861fdba354-ca-trust-extracted\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-image-registry-private-configuration\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cc40c6d-de39-4283-bb66-f6aa7bb99ac0-cert\") pod \"ingress-canary-vt9w5\" (UID: \"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0\") " pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.941634 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.941634 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f183322-bfbb-4763-b013-72e012cfa318-tmp-dir\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.941634 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvk9p\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-kube-api-access-pvk9p\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.941634 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:55.941537 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:18:55.941634 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:55.941594 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:56.441574628 +0000 UTC m=+38.106816449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:18:55.941945 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.941924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-certificates\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.942053 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.942027 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6f183322-bfbb-4763-b013-72e012cfa318-config-volume\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.942771 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.942719 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-trusted-ca\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.945716 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.945696 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-installation-pull-secrets\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.945804 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.945703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7cc40c6d-de39-4283-bb66-f6aa7bb99ac0-cert\") pod \"ingress-canary-vt9w5\" (UID: \"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0\") " pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.945804 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.945775 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-image-registry-private-configuration\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.952494 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.952469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkmt\" (UniqueName: \"kubernetes.io/projected/6f183322-bfbb-4763-b013-72e012cfa318-kube-api-access-7kkmt\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:55.952597 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.952473 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvk9p\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-kube-api-access-pvk9p\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:55.952658 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.952613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzvr\" (UniqueName: \"kubernetes.io/projected/7cc40c6d-de39-4283-bb66-f6aa7bb99ac0-kube-api-access-5bzvr\") pod \"ingress-canary-vt9w5\" (UID: \"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0\") " pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:55.953019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:55.953002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-bound-sa-token\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:56.025173 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.025149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vt9w5" Jul 28 14:18:56.233972 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.233746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vt9w5"] Jul 28 14:18:56.238243 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:18:56.238219 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc40c6d_de39_4283_bb66_f6aa7bb99ac0.slice/crio-8915778768ece043ab889f984bd2a2077dac214de2016b0ac88a356570acfb9c WatchSource:0}: Error finding container 8915778768ece043ab889f984bd2a2077dac214de2016b0ac88a356570acfb9c: Status 404 returned error can't find the container with id 8915778768ece043ab889f984bd2a2077dac214de2016b0ac88a356570acfb9c Jul 28 14:18:56.445998 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.445968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:56.446103 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.446021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:56.446159 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:56.446135 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:18:56.446213 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:56.446206 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:57.4461894 +0000 UTC m=+39.111431205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:18:56.446278 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:56.446142 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:18:56.446278 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:56.446242 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:18:56.446363 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:56.446315 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:57.446296097 +0000 UTC m=+39.111537937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:18:56.883052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.883025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:18:56.883052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.883040 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:18:56.883619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.883025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:18:56.886146 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.886119 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 14:18:56.886300 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.886279 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 14:18:56.886375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.886281 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 14:18:56.886375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.886286 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 14:18:56.886375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.886318 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4vsp\"" Jul 28 14:18:56.886375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:56.886360 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4gd6j\"" Jul 28 14:18:57.069793 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:57.069743 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vt9w5" event={"ID":"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0","Type":"ContainerStarted","Data":"8915778768ece043ab889f984bd2a2077dac214de2016b0ac88a356570acfb9c"} Jul 28 14:18:57.454867 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:57.454824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:57.455050 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:57.454900 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:57.455050 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:57.454992 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:18:57.455050 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:57.455040 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:18:57.455167 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:57.455053 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:18:57.455167 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:57.455080 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:59.455059499 +0000 UTC m=+41.120301305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:18:57.455167 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:57.455108 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:18:59.455093386 +0000 UTC m=+41.120335202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:18:59.472126 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:59.472081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:18:59.472547 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:18:59.472134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:18:59.472547 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:59.472241 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:18:59.472547 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:59.472247 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:18:59.472547 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:59.472252 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:18:59.472547 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:59.472309 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:03.472293985 +0000 UTC m=+45.137535786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:18:59.472547 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:18:59.472381 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:03.472359717 +0000 UTC m=+45.137601546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:19:00.077983 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:00.077951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vt9w5" event={"ID":"7cc40c6d-de39-4283-bb66-f6aa7bb99ac0","Type":"ContainerStarted","Data":"8a090a9cb3bdba8f6fbc49e059566c4b35776fc6095e2a7f7d70fccdf04ba4f7"} Jul 28 14:19:00.098247 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:00.096783 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vt9w5" podStartSLOduration=2.04633313 podStartE2EDuration="5.096761687s" totalImagesPullingTime="3.050428557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:00.095591138 +0000 UTC m=+41.760832966" watchObservedRunningTime="2026-07-28 14:19:00.096761687 +0000 UTC m=+41.762003514" Jul 28 14:19:03.503495 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:03.503278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:19:03.503940 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:03.503430 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:19:03.503940 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:03.503539 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:19:03.503940 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:03.503609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:19:03.503940 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:03.503623 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:11.503603781 +0000 UTC m=+53.168845606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:19:03.503940 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:03.503707 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:19:03.503940 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:03.503753 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:11.503740982 +0000 UTC m=+53.168982801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:19:03.553374 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:03.553329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:03.633964 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:03.633931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:04.834173 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:04.834132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:04.913614 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:04.913569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:05.291436 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:05.291355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:05.368036 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:05.368000 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:07.981788 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:07.981748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:11.562708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:11.562653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:19:11.563171 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:11.562779 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:19:11.563171 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:11.562814 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:19:11.563171 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:11.562831 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:19:11.563171 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:11.562872 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:19:11.563171 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:11.562897 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:27.562878269 +0000 UTC m=+69.228120087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:19:11.563171 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:11.562913 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:27.562905795 +0000 UTC m=+69.228147597 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:19:13.204910 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:13.204857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:13.302806 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:13.302766 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:13.548725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:13.548692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:13.648719 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:13.648687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:13.746395 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:13.746360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:13.846193 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:13.846102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:14.168170 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:14.168101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:15.323383 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:15.323339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:15.448374 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:15.448351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:15.698455 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:15.698396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:15.804825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:15.804798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:17.102589 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:17.102559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:18.547165 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:18.547126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:20.053208 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:20.053175 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brkdl" Jul 28 14:19:20.212604 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:20.212572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:20.281419 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:20.281389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:20.356636 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:20.356566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:22.238476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.238440 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng"] Jul 28 14:19:22.267405 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.267383 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-77859b6fc7-vhkkx"] Jul 28 14:19:22.267535 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.267495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.270100 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.270072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jfprg\"" Jul 28 14:19:22.270233 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.270112 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 14:19:22.270915 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.270879 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 14:19:22.271211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.271190 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 14:19:22.271324 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.271196 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 14:19:22.292090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.292064 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng"] Jul 28 14:19:22.292090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.292094 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77859b6fc7-vhkkx"] Jul 28 14:19:22.292207 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.292158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.294269 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294250 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 14:19:22.294349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294257 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 14:19:22.294414 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 14:19:22.294569 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294554 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 14:19:22.294707 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294691 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 14:19:22.294785 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294754 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 14:19:22.294785 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.294774 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vrbb9\"" Jul 28 14:19:22.331868 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.331850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wc5\" (UniqueName: \"kubernetes.io/projected/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-kube-api-access-h2wc5\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.331971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.331888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.331971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.331908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-default-certificate\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.331971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.331923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2lt8\" (UniqueName: \"kubernetes.io/projected/648fe082-e7c4-40bf-a820-b594afd97f9c-kube-api-access-b2lt8\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.331971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.331942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.332101 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.332010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.332101 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.332063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.332164 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.332098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-stats-auth\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.369077 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.369055 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-qml4d"] Jul 28 14:19:22.393440 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.393423 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-qml4d"] Jul 28 14:19:22.393515 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.393490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.396816 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.396776 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 14:19:22.396914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.396791 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 14:19:22.396914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.396797 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 14:19:22.396914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.396893 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 14:19:22.397270 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.397256 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n8nj6\"" Jul 28 14:19:22.403554 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.403538 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 14:19:22.432755 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2lt8\" (UniqueName: \"kubernetes.io/projected/648fe082-e7c4-40bf-a820-b594afd97f9c-kube-api-access-b2lt8\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.432838 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e16a89f6-30f6-4e0d-9823-df204fb02fdd-tmp\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.432838 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a89f6-30f6-4e0d-9823-df204fb02fdd-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.432838 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.432939 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.432939 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.432877 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 14:19:22.432939 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.432939 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-stats-auth\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.432939 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.432931 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:22.932914249 +0000 UTC m=+64.598156074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : secret "router-metrics-certs-default" not found Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.432959 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e16a89f6-30f6-4e0d-9823-df204fb02fdd-serving-cert\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.432969 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.432998 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:22.932981387 +0000 UTC m=+64.598223218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : configmap references non-existent config key: service-ca.crt Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptcn2\" (UniqueName: \"kubernetes.io/projected/e16a89f6-30f6-4e0d-9823-df204fb02fdd-kube-api-access-ptcn2\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.433095 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls podName:0dba08a3-6d95-4cf5-aaf6-6f4549eb1531 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:22.933082362 +0000 UTC m=+64.598324169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9x8ng" (UID: "0dba08a3-6d95-4cf5-aaf6-6f4549eb1531") : secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-default-certificate\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.433161 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e16a89f6-30f6-4e0d-9823-df204fb02fdd-snapshots\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.433468 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433190 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a89f6-30f6-4e0d-9823-df204fb02fdd-service-ca-bundle\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.433468 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wc5\" (UniqueName: \"kubernetes.io/projected/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-kube-api-access-h2wc5\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.433773 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.433756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.436595 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.436575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-stats-auth\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.436692 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.436666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-default-certificate\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.443484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.443457 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wc5\" (UniqueName: \"kubernetes.io/projected/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-kube-api-access-h2wc5\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.443782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.443766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2lt8\" (UniqueName: \"kubernetes.io/projected/648fe082-e7c4-40bf-a820-b594afd97f9c-kube-api-access-b2lt8\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.488940 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.488874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:22.533941 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.533916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e16a89f6-30f6-4e0d-9823-df204fb02fdd-tmp\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534023 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.533947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a89f6-30f6-4e0d-9823-df204fb02fdd-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534023 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e16a89f6-30f6-4e0d-9823-df204fb02fdd-serving-cert\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534100 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptcn2\" (UniqueName: \"kubernetes.io/projected/e16a89f6-30f6-4e0d-9823-df204fb02fdd-kube-api-access-ptcn2\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534263 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e16a89f6-30f6-4e0d-9823-df204fb02fdd-snapshots\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534309 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a89f6-30f6-4e0d-9823-df204fb02fdd-service-ca-bundle\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534309 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534301 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e16a89f6-30f6-4e0d-9823-df204fb02fdd-tmp\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534711 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a89f6-30f6-4e0d-9823-df204fb02fdd-service-ca-bundle\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534822 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e16a89f6-30f6-4e0d-9823-df204fb02fdd-snapshots\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.534886 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.534825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a89f6-30f6-4e0d-9823-df204fb02fdd-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.536057 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.536036 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e16a89f6-30f6-4e0d-9823-df204fb02fdd-serving-cert\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.543743 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.543725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptcn2\" (UniqueName: \"kubernetes.io/projected/e16a89f6-30f6-4e0d-9823-df204fb02fdd-kube-api-access-ptcn2\") pod \"insights-operator-5885fdb759-qml4d\" (UID: \"e16a89f6-30f6-4e0d-9823-df204fb02fdd\") " pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.556964 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.556942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:22.701265 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.701244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-qml4d" Jul 28 14:19:22.867513 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.867486 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-qml4d"] Jul 28 14:19:22.870341 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:22.870318 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16a89f6_30f6_4e0d_9823_df204fb02fdd.slice/crio-c01cac8d49302056355858b2d314e673694e963005aa10aab3ae5d90ab4cb05d WatchSource:0}: Error finding container c01cac8d49302056355858b2d314e673694e963005aa10aab3ae5d90ab4cb05d: Status 404 returned error can't find the container with id c01cac8d49302056355858b2d314e673694e963005aa10aab3ae5d90ab4cb05d Jul 28 14:19:22.937168 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.937142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.937249 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.937172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:22.937249 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:22.937202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:22.937326 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.937310 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 14:19:22.937326 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.937322 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:22.937396 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.937329 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:23.937311063 +0000 UTC m=+65.602552881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : configmap references non-existent config key: service-ca.crt Jul 28 14:19:22.937396 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.937360 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:23.937344045 +0000 UTC m=+65.602585847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : secret "router-metrics-certs-default" not found Jul 28 14:19:22.937396 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:22.937376 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls podName:0dba08a3-6d95-4cf5-aaf6-6f4549eb1531 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:23.937368639 +0000 UTC m=+65.602610442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9x8ng" (UID: "0dba08a3-6d95-4cf5-aaf6-6f4549eb1531") : secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:23.132947 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:23.132921 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qml4d" event={"ID":"e16a89f6-30f6-4e0d-9823-df204fb02fdd","Type":"ContainerStarted","Data":"c01cac8d49302056355858b2d314e673694e963005aa10aab3ae5d90ab4cb05d"} Jul 28 14:19:23.942575 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:23.942545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:23.942575 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:23.942577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:23.943049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:23.942629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:23.943049 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:23.942764 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:25.942741866 +0000 UTC m=+67.607983669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : configmap references non-existent config key: service-ca.crt Jul 28 14:19:23.943049 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:23.942780 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:23.943049 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:23.942847 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls podName:0dba08a3-6d95-4cf5-aaf6-6f4549eb1531 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:25.942831456 +0000 UTC m=+67.608073260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9x8ng" (UID: "0dba08a3-6d95-4cf5-aaf6-6f4549eb1531") : secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:23.943049 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:23.942846 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 14:19:23.943049 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:23.942885 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:25.942875427 +0000 UTC m=+67.608117232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : secret "router-metrics-certs-default" not found Jul 28 14:19:24.546928 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.546891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:19:24.549642 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.549620 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 14:19:24.560270 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.560250 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a38c15d-bd62-4e10-8c87-120662bd95f7-original-pull-secret\") pod \"global-pull-secret-syncer-fgmfx\" (UID: \"5a38c15d-bd62-4e10-8c87-120662bd95f7\") " pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:19:24.647607 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.647564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:19:24.647752 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.647655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:19:24.650496 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.650472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 14:19:24.650607 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.650510 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 14:19:24.658288 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:24.658268 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 14:19:24.658394 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:24.658340 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs podName:f0fded26-1385-4fb1-93df-dea08d2b474c nodeName:}" failed. No retries permitted until 2026-07-28 14:20:28.658319485 +0000 UTC m=+130.323561286 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs") pod "network-metrics-daemon-8sjnc" (UID: "f0fded26-1385-4fb1-93df-dea08d2b474c") : secret "metrics-daemon-secret" not found Jul 28 14:19:24.660566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.660539 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 14:19:24.671723 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.671703 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgml\" (UniqueName: \"kubernetes.io/projected/ecb6e983-1363-43cf-aa9c-0626a42d476b-kube-api-access-6kgml\") pod \"network-check-target-fpqnz\" (UID: \"ecb6e983-1363-43cf-aa9c-0626a42d476b\") " pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:19:24.800267 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.800224 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fgmfx" Jul 28 14:19:24.807232 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.807215 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4vsp\"" Jul 28 14:19:24.815370 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.815350 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:19:24.941378 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.941349 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fgmfx"] Jul 28 14:19:24.944753 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:24.944728 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a38c15d_bd62_4e10_8c87_120662bd95f7.slice/crio-c5dbdf4f68c95d7e21d35886f1cf36f666bfc2abe673b8dad8956a5e2a53da6b WatchSource:0}: Error finding container c5dbdf4f68c95d7e21d35886f1cf36f666bfc2abe673b8dad8956a5e2a53da6b: Status 404 returned error can't find the container with id c5dbdf4f68c95d7e21d35886f1cf36f666bfc2abe673b8dad8956a5e2a53da6b Jul 28 14:19:24.957225 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:24.957202 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fpqnz"] Jul 28 14:19:24.960092 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:24.960075 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb6e983_1363_43cf_aa9c_0626a42d476b.slice/crio-b96c7392d22cb8bc3c5ae9e39d4a4d657df58582f1274c08131e27bd793f4f97 WatchSource:0}: Error finding container b96c7392d22cb8bc3c5ae9e39d4a4d657df58582f1274c08131e27bd793f4f97: Status 404 returned error can't find the container with id b96c7392d22cb8bc3c5ae9e39d4a4d657df58582f1274c08131e27bd793f4f97 Jul 28 14:19:25.138423 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.138357 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fpqnz" event={"ID":"ecb6e983-1363-43cf-aa9c-0626a42d476b","Type":"ContainerStarted","Data":"b96c7392d22cb8bc3c5ae9e39d4a4d657df58582f1274c08131e27bd793f4f97"} Jul 28 14:19:25.139567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.139547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fgmfx" event={"ID":"5a38c15d-bd62-4e10-8c87-120662bd95f7","Type":"ContainerStarted","Data":"c5dbdf4f68c95d7e21d35886f1cf36f666bfc2abe673b8dad8956a5e2a53da6b"} Jul 28 14:19:25.140865 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.140847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qml4d" event={"ID":"e16a89f6-30f6-4e0d-9823-df204fb02fdd","Type":"ContainerStarted","Data":"47e572d63fc39a3fe0998171b50357d8c18d3c0cc13fb2e92eb1f0d2baaaffeb"} Jul 28 14:19:25.161827 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.161786 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-qml4d" podStartSLOduration=1.304030107 podStartE2EDuration="3.161770247s" totalImagesPullingTime="1.85774014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:25.161754517 +0000 UTC m=+66.826996340" watchObservedRunningTime="2026-07-28 14:19:25.161770247 +0000 UTC m=+66.827012072" Jul 28 14:19:25.275081 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.275060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:25.959580 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.958908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:25.959580 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.958950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:25.959580 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:25.958990 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:25.959580 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:25.959182 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:25.959580 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:25.959246 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls podName:0dba08a3-6d95-4cf5-aaf6-6f4549eb1531 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:29.959228089 +0000 UTC m=+71.624469913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9x8ng" (UID: "0dba08a3-6d95-4cf5-aaf6-6f4549eb1531") : secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:25.959580 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:25.959549 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:29.95953208 +0000 UTC m=+71.624773892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : configmap references non-existent config key: service-ca.crt Jul 28 14:19:25.960302 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:25.959617 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 14:19:25.960302 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:25.959654 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:29.959640853 +0000 UTC m=+71.624882659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : secret "router-metrics-certs-default" not found Jul 28 14:19:26.215729 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.215589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:26.282650 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.282623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:26.670326 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.670292 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bm8mr"] Jul 28 14:19:26.673435 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.673411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.684554 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.684530 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 14:19:26.685342 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.685317 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kr6cm\"" Jul 28 14:19:26.685800 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.685780 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 14:19:26.705303 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.705239 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bm8mr"] Jul 28 14:19:26.728348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.728316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:26.767097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.767071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.767200 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.767122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6aec9c3a-0460-4712-b808-9665f7eb3d39-crio-socket\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.767200 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.767194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.767322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.767224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89k2t\" (UniqueName: \"kubernetes.io/projected/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-api-access-89k2t\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.767322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.767267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6aec9c3a-0460-4712-b808-9665f7eb3d39-data-volume\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.794086 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.794056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:26.867900 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.867869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.868053 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.867914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2t\" (UniqueName: \"kubernetes.io/projected/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-api-access-89k2t\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.868053 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.867964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6aec9c3a-0460-4712-b808-9665f7eb3d39-data-volume\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.868175 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:26.868050 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 14:19:26.868175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.868082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.868175 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:26.868122 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:27.36810141 +0000 UTC m=+69.033343213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : secret "insights-runtime-extractor-tls" not found Jul 28 14:19:26.868175 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:26.868151 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 14:19:26.868398 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.868181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6aec9c3a-0460-4712-b808-9665f7eb3d39-crio-socket\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.868398 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:26.868203 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:27.368188059 +0000 UTC m=+69.033429875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : configmap "kube-rbac-proxy" not found Jul 28 14:19:26.868398 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.868272 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6aec9c3a-0460-4712-b808-9665f7eb3d39-crio-socket\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.868398 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.868327 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6aec9c3a-0460-4712-b808-9665f7eb3d39-data-volume\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:26.879336 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:26.879306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k2t\" (UniqueName: \"kubernetes.io/projected/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-api-access-89k2t\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:27.372988 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:27.372952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:27.373456 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:27.373054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:27.373456 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.373198 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 14:19:27.373456 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.373257 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:28.373238057 +0000 UTC m=+70.038479866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : secret "insights-runtime-extractor-tls" not found Jul 28 14:19:27.373639 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.373608 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 14:19:27.373713 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.373656 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:28.373644249 +0000 UTC m=+70.038886049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : configmap "kube-rbac-proxy" not found Jul 28 14:19:27.574768 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:27.574738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:19:27.574955 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:27.574808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:19:27.574955 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.574901 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 14:19:27.574955 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.574922 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cbdbcdff6-gvqcz: secret "image-registry-tls" not found Jul 28 14:19:27.574955 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.574947 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 14:19:27.575140 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.574982 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls podName:917ce53a-a02b-4fa9-9c8d-37861fdba354 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:59.574964306 +0000 UTC m=+101.240206113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls") pod "image-registry-5cbdbcdff6-gvqcz" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354") : secret "image-registry-tls" not found Jul 28 14:19:27.575140 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:27.575002 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls podName:6f183322-bfbb-4763-b013-72e012cfa318 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:59.574990908 +0000 UTC m=+101.240232715 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls") pod "dns-default-7qlhl" (UID: "6f183322-bfbb-4763-b013-72e012cfa318") : secret "dns-default-metrics-tls" not found Jul 28 14:19:28.381688 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:28.381641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:28.382185 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:28.381742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:28.382185 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:28.381750 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 14:19:28.382185 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:28.381860 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:30.381840451 +0000 UTC m=+72.047082255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : configmap "kube-rbac-proxy" not found Jul 28 14:19:28.382185 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:28.381918 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 14:19:28.382185 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:28.381993 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:30.381972184 +0000 UTC m=+72.047214006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : secret "insights-runtime-extractor-tls" not found Jul 28 14:19:28.458825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:28.458801 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bs7j4_f842320b-70c6-466f-9e1c-fcf38d578666/dns-node-resolver/0.log" Jul 28 14:19:29.223447 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.223381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:29.297813 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.297774 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:29.420655 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.420621 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:29.458542 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.458523 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ttdk2_a4f710d1-1eaa-45e8-8853-4bdc020989a0/node-ca/0.log" Jul 28 14:19:29.487654 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.487596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:29.995356 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.995314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:29.995356 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.995360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:29.995562 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:29.995487 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 14:19:29.995562 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:29.995510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:29.995633 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:29.995565 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:37.995544775 +0000 UTC m=+79.660786617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : secret "router-metrics-certs-default" not found Jul 28 14:19:29.995633 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:29.995605 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:29.995633 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:29.995612 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:37.995597418 +0000 UTC m=+79.660839219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : configmap references non-existent config key: service-ca.crt Jul 28 14:19:29.995771 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:29.995644 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls podName:0dba08a3-6d95-4cf5-aaf6-6f4549eb1531 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:37.995632178 +0000 UTC m=+79.660873984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9x8ng" (UID: "0dba08a3-6d95-4cf5-aaf6-6f4549eb1531") : secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:30.157262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.157234 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fpqnz" event={"ID":"ecb6e983-1363-43cf-aa9c-0626a42d476b","Type":"ContainerStarted","Data":"0a83ad43d169ccfb6c83db7410b953b97119c1d370d75f4b45d388cb1c03cfc5"} Jul 28 14:19:30.157415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.157369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:19:30.158955 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.158929 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fgmfx" event={"ID":"5a38c15d-bd62-4e10-8c87-120662bd95f7","Type":"ContainerStarted","Data":"f27409839f4ce37cf91e3850afc0bafc200c9a1d16e0260c283bf11760a0504f"} Jul 28 14:19:30.175057 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.175005 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fpqnz" podStartSLOduration=67.810419376 podStartE2EDuration="1m12.174987459s" totalImagesPullingTime="4.364568083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:30.174489564 +0000 UTC m=+71.839731388" watchObservedRunningTime="2026-07-28 14:19:30.174987459 +0000 UTC m=+71.840229282" Jul 28 14:19:30.397503 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.397476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:30.397609 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.397539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:30.397609 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:30.397590 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 14:19:30.397713 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:30.397648 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:34.39763307 +0000 UTC m=+76.062874872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : configmap "kube-rbac-proxy" not found Jul 28 14:19:30.397713 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:30.397650 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 14:19:30.397785 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:30.397718 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:34.397701512 +0000 UTC m=+76.062943370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : secret "insights-runtime-extractor-tls" not found Jul 28 14:19:30.458094 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.458076 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vt9w5_7cc40c6d-de39-4283-bb66-f6aa7bb99ac0/serve-healthcheck-canary/0.log" Jul 28 14:19:30.715630 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.715571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:30.817032 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:30.817007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:34.429102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:34.429064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:34.429517 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:34.429140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:34.429517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:34.429216 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 14:19:34.429517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:34.429255 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 14:19:34.429517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:34.429274 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:42.429258914 +0000 UTC m=+84.094500716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : configmap "kube-rbac-proxy" not found Jul 28 14:19:34.429517 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:34.429317 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:42.4293018 +0000 UTC m=+84.094543601 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : secret "insights-runtime-extractor-tls" not found Jul 28 14:19:36.212379 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:36.212341 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:36.355748 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:36.355713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:36.452117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:36.452086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:36.568843 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:36.568820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:37.258348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:37.258315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:37.329568 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:37.329535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:37.665271 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:37.665237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:38.057657 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:38.057632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:38.057800 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:38.057666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:38.057800 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:38.057721 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:38.057919 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:38.057795 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 14:19:38.057919 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:38.057823 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:54.057805969 +0000 UTC m=+95.723047771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : configmap references non-existent config key: service-ca.crt Jul 28 14:19:38.057919 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:38.057850 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs podName:648fe082-e7c4-40bf-a820-b594afd97f9c nodeName:}" failed. No retries permitted until 2026-07-28 14:19:54.057837503 +0000 UTC m=+95.723079303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs") pod "router-default-77859b6fc7-vhkkx" (UID: "648fe082-e7c4-40bf-a820-b594afd97f9c") : secret "router-metrics-certs-default" not found Jul 28 14:19:38.057919 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:38.057894 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:38.058236 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:38.057961 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls podName:0dba08a3-6d95-4cf5-aaf6-6f4549eb1531 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:54.0579448 +0000 UTC m=+95.723186635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-9x8ng" (UID: "0dba08a3-6d95-4cf5-aaf6-6f4549eb1531") : secret "cluster-monitoring-operator-tls" not found Jul 28 14:19:38.474998 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:38.474906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:38.536961 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:38.536924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:40.066807 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:40.066767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:40.173440 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:40.173394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:40.580264 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:40.580229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:41.425169 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:41.425133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:41.635886 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:41.635859 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:41.725209 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:41.725145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:41.825915 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:41.825884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:41.925609 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:41.925579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:42.126723 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:42.126671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:42.488995 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:42.488936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:42.489293 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:42.489012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:42.489293 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:42.489099 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 14:19:42.489293 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:19:42.489163 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls podName:6aec9c3a-0460-4712-b808-9665f7eb3d39 nodeName:}" failed. No retries permitted until 2026-07-28 14:19:58.489147054 +0000 UTC m=+100.154388855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bm8mr" (UID: "6aec9c3a-0460-4712-b808-9665f7eb3d39") : secret "insights-runtime-extractor-tls" not found Jul 28 14:19:42.489534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:42.489512 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6aec9c3a-0460-4712-b808-9665f7eb3d39-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:42.927283 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:42.927252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:43.338012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:43.337981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:43.723797 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:43.723734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:43.795281 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:43.795251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:44.122387 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:44.122368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:44.341845 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:44.341816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:44.724260 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:44.724230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:45.119280 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:45.119255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:45.220021 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:45.219992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:45.731913 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:45.731877 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:45.829287 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:45.829254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:46.163224 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:46.163195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:46.324926 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:46.324904 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:46.425775 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:46.425719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:46.921156 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:46.921126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:47.534032 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:47.533994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:47.722306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:47.722282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:47.964061 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:47.964034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:48.020189 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:48.020162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:48.124799 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:48.124773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:49.328075 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:49.328035 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:49.532790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:49.532762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:49.727704 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:49.727608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:49.935893 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:49.935863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:50.207664 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:50.207630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:50.308754 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:50.308723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:50.404147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:50.404123 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:51.128068 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:51.128037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:51.341163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:51.341127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:51.738548 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:51.738516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:51.928081 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:51.928053 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:52.126039 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:52.126012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:52.190879 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:52.190846 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:54.079901 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.079868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:54.079901 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.079902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:54.080470 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.079930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:54.080544 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.080520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/648fe082-e7c4-40bf-a820-b594afd97f9c-service-ca-bundle\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:54.082600 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.082565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/648fe082-e7c4-40bf-a820-b594afd97f9c-metrics-certs\") pod \"router-default-77859b6fc7-vhkkx\" (UID: \"648fe082-e7c4-40bf-a820-b594afd97f9c\") " pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:54.082763 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.082632 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dba08a3-6d95-4cf5-aaf6-6f4549eb1531-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-9x8ng\" (UID: \"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:54.100440 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.100417 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:54.224031 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.223979 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fgmfx" podStartSLOduration=90.839177092 podStartE2EDuration="1m35.223958574s" totalImagesPullingTime="4.384781482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:30.190191994 +0000 UTC m=+71.855433819" watchObservedRunningTime="2026-07-28 14:19:54.223958574 +0000 UTC m=+95.889200412" Jul 28 14:19:54.225276 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.225253 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-77859b6fc7-vhkkx"] Jul 28 14:19:54.228541 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:54.228507 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648fe082_e7c4_40bf_a820_b594afd97f9c.slice/crio-c9b37c2e8466662685ad985a9b9c6569af9eda516ecb1a371dd1af220bbb4034 WatchSource:0}: Error finding container c9b37c2e8466662685ad985a9b9c6569af9eda516ecb1a371dd1af220bbb4034: Status 404 returned error can't find the container with id c9b37c2e8466662685ad985a9b9c6569af9eda516ecb1a371dd1af220bbb4034 Jul 28 14:19:54.376669 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.376609 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" Jul 28 14:19:54.496620 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.496593 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng"] Jul 28 14:19:54.499171 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:54.499146 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dba08a3_6d95_4cf5_aaf6_6f4549eb1531.slice/crio-6032dc5880d7b89bb42b98f188b18893c2d55f94206207ef73aed4e69c1200d5 WatchSource:0}: Error finding container 6032dc5880d7b89bb42b98f188b18893c2d55f94206207ef73aed4e69c1200d5: Status 404 returned error can't find the container with id 6032dc5880d7b89bb42b98f188b18893c2d55f94206207ef73aed4e69c1200d5 Jul 28 14:19:54.751250 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.751169 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:54.869871 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:54.869844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:55.219403 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.219376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" event={"ID":"648fe082-e7c4-40bf-a820-b594afd97f9c","Type":"ContainerStarted","Data":"ad585c50ceda85a1108519a74013a43ec0f0b89365837171166b083b992ab109"} Jul 28 14:19:55.219811 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.219433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" event={"ID":"648fe082-e7c4-40bf-a820-b594afd97f9c","Type":"ContainerStarted","Data":"c9b37c2e8466662685ad985a9b9c6569af9eda516ecb1a371dd1af220bbb4034"} Jul 28 14:19:55.220665 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.220644 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" event={"ID":"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531","Type":"ContainerStarted","Data":"6032dc5880d7b89bb42b98f188b18893c2d55f94206207ef73aed4e69c1200d5"} Jul 28 14:19:55.241258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.241208 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" podStartSLOduration=33.241194505 podStartE2EDuration="33.241194505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:55.241071466 +0000 UTC m=+96.906313290" watchObservedRunningTime="2026-07-28 14:19:55.241194505 +0000 UTC m=+96.906436328" Jul 28 14:19:55.262275 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.262252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:55.444443 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.444412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:55.661699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.661638 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:55.739003 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.738973 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:55.875341 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.875311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:55.940569 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:55.940502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:56.101578 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.101551 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:56.104409 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.104388 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:56.224618 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.224549 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:56.225846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.225827 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-77859b6fc7-vhkkx" Jul 28 14:19:56.690465 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.690381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:56.861264 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.861182 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:56.964452 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:56.964422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:57.061193 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.061166 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:57.222521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.222458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:57.228893 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.228865 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" event={"ID":"0dba08a3-6d95-4cf5-aaf6-6f4549eb1531","Type":"ContainerStarted","Data":"90f3b99097771fbdcb98d3c8730af4f30bd8be7a3f0755e43eae4640c76ac659"} Jul 28 14:19:57.245307 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.245266 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-9x8ng" podStartSLOduration=33.182678293 podStartE2EDuration="35.245253097s" totalImagesPullingTime="2.062574804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:19:57.245057413 +0000 UTC m=+98.910299237" watchObservedRunningTime="2026-07-28 14:19:57.245253097 +0000 UTC m=+98.910494920" Jul 28 14:19:57.280917 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.280897 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:57.664178 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.664146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:57.727717 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:57.727692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:58.231499 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:58.231461 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:58.302598 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:58.302574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:58.516705 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:58.516626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:58.519009 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:58.518977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6aec9c3a-0460-4712-b808-9665f7eb3d39-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bm8mr\" (UID: \"6aec9c3a-0460-4712-b808-9665f7eb3d39\") " pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:58.785128 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:58.785077 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bm8mr" Jul 28 14:19:58.912987 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:58.912960 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bm8mr"] Jul 28 14:19:58.915668 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:58.915636 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aec9c3a_0460_4712_b808_9665f7eb3d39.slice/crio-dd69a5440bdc6bcbe9b0d5bcb39af779da79f96f404484f0563ca9df4114f7e3 WatchSource:0}: Error finding container dd69a5440bdc6bcbe9b0d5bcb39af779da79f96f404484f0563ca9df4114f7e3: Status 404 returned error can't find the container with id dd69a5440bdc6bcbe9b0d5bcb39af779da79f96f404484f0563ca9df4114f7e3 Jul 28 14:19:59.237541 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.237504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm8mr" event={"ID":"6aec9c3a-0460-4712-b808-9665f7eb3d39","Type":"ContainerStarted","Data":"dd69a5440bdc6bcbe9b0d5bcb39af779da79f96f404484f0563ca9df4114f7e3"} Jul 28 14:19:59.528234 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.528163 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-p9h77"] Jul 28 14:19:59.531415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.531390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.534562 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.534542 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 28 14:19:59.534659 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.534571 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 28 14:19:59.534659 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.534597 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8dj6f\"" Jul 28 14:19:59.546021 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.545965 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-p9h77"] Jul 28 14:19:59.562695 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.562644 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh"] Jul 28 14:19:59.565851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.565830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.568790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.568389 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 14:19:59.568943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.568790 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 14:19:59.568943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.568929 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 28 14:19:59.569301 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.569262 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 14:19:59.579880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.579827 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh"] Jul 28 14:19:59.628102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.627190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:19:59.628102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.627247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b586543-0933-4f68-8be0-fce454e73554-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-p9h77\" (UID: \"4b586543-0933-4f68-8be0-fce454e73554\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.628102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.627278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b586543-0933-4f68-8be0-fce454e73554-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-p9h77\" (UID: \"4b586543-0933-4f68-8be0-fce454e73554\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.628102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.627360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:19:59.628102 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.627983 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8"] Jul 28 14:19:59.630933 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.630848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"image-registry-5cbdbcdff6-gvqcz\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:19:59.631187 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.631172 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" Jul 28 14:19:59.632309 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.632269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f183322-bfbb-4763-b013-72e012cfa318-metrics-tls\") pod \"dns-default-7qlhl\" (UID: \"6f183322-bfbb-4763-b013-72e012cfa318\") " pod="openshift-dns/dns-default-7qlhl" Jul 28 14:19:59.634686 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.634499 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sznln\"" Jul 28 14:19:59.646416 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.643988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8"] Jul 28 14:19:59.651323 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.651258 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6f28j\"" Jul 28 14:19:59.657656 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.657632 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7qlhl" Jul 28 14:19:59.731355 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.731284 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c1a2e61-3056-4e8b-8118-d17992749b0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.731501 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.731373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjp5\" (UniqueName: \"kubernetes.io/projected/8c1a2e61-3056-4e8b-8118-d17992749b0a-kube-api-access-7mjp5\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.731501 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.731422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b586543-0933-4f68-8be0-fce454e73554-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-p9h77\" (UID: \"4b586543-0933-4f68-8be0-fce454e73554\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.731501 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.731447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c1a2e61-3056-4e8b-8118-d17992749b0a-tmp\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.731501 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.731473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b586543-0933-4f68-8be0-fce454e73554-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-p9h77\" (UID: \"4b586543-0933-4f68-8be0-fce454e73554\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.731739 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.731536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4msm\" (UniqueName: \"kubernetes.io/projected/a2b9f21d-0e5d-48fd-87b1-f8d66f294106-kube-api-access-x4msm\") pod \"network-check-source-7d88cb55d8-kp7k8\" (UID: \"a2b9f21d-0e5d-48fd-87b1-f8d66f294106\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" Jul 28 14:19:59.732350 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.732321 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b586543-0933-4f68-8be0-fce454e73554-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-p9h77\" (UID: \"4b586543-0933-4f68-8be0-fce454e73554\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.736075 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.735033 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b586543-0933-4f68-8be0-fce454e73554-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-p9h77\" (UID: \"4b586543-0933-4f68-8be0-fce454e73554\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.754651 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.754607 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:59.827535 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.827337 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7qlhl"] Jul 28 14:19:59.831853 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:19:59.831821 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f183322_bfbb_4763_b013_72e012cfa318.slice/crio-9418b9d643580f60474bc1efa0ffe4cd6fbb7957ddbea7e1692576dbb54bbbcb WatchSource:0}: Error finding container 9418b9d643580f60474bc1efa0ffe4cd6fbb7957ddbea7e1692576dbb54bbbcb: Status 404 returned error can't find the container with id 9418b9d643580f60474bc1efa0ffe4cd6fbb7957ddbea7e1692576dbb54bbbcb Jul 28 14:19:59.831962 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.831879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c1a2e61-3056-4e8b-8118-d17992749b0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.831962 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.831920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjp5\" (UniqueName: \"kubernetes.io/projected/8c1a2e61-3056-4e8b-8118-d17992749b0a-kube-api-access-7mjp5\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.832083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.831961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c1a2e61-3056-4e8b-8118-d17992749b0a-tmp\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.832083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.832017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4msm\" (UniqueName: \"kubernetes.io/projected/a2b9f21d-0e5d-48fd-87b1-f8d66f294106-kube-api-access-x4msm\") pod \"network-check-source-7d88cb55d8-kp7k8\" (UID: \"a2b9f21d-0e5d-48fd-87b1-f8d66f294106\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" Jul 28 14:19:59.832865 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.832720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c1a2e61-3056-4e8b-8118-d17992749b0a-tmp\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.838737 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.838546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8c1a2e61-3056-4e8b-8118-d17992749b0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.843500 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.843481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4msm\" (UniqueName: \"kubernetes.io/projected/a2b9f21d-0e5d-48fd-87b1-f8d66f294106-kube-api-access-x4msm\") pod \"network-check-source-7d88cb55d8-kp7k8\" (UID: \"a2b9f21d-0e5d-48fd-87b1-f8d66f294106\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" Jul 28 14:19:59.844071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.844048 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjp5\" (UniqueName: \"kubernetes.io/projected/8c1a2e61-3056-4e8b-8118-d17992749b0a-kube-api-access-7mjp5\") pod \"klusterlet-addon-workmgr-f8944f5dd-cchnh\" (UID: \"8c1a2e61-3056-4e8b-8118-d17992749b0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.846306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.846284 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" Jul 28 14:19:59.853431 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.853379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:59.878355 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.878314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:19:59.918943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.918903 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lj7bw\"" Jul 28 14:19:59.927826 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.927584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:19:59.957661 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.954054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:19:59.957661 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:19:59.957240 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" Jul 28 14:20:00.019718 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.019292 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-p9h77"] Jul 28 14:20:00.032732 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:00.032595 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b586543_0933_4f68_8be0_fce454e73554.slice/crio-fc0586e3d1eeab39926843c7a724271bd91b40b5a9938867e7e3fce581d894a4 WatchSource:0}: Error finding container fc0586e3d1eeab39926843c7a724271bd91b40b5a9938867e7e3fce581d894a4: Status 404 returned error can't find the container with id fc0586e3d1eeab39926843c7a724271bd91b40b5a9938867e7e3fce581d894a4 Jul 28 14:20:00.061161 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.060919 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:00.072119 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.072078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh"] Jul 28 14:20:00.086387 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:00.086242 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1a2e61_3056_4e8b_8118_d17992749b0a.slice/crio-7a7eafabaca1fc123ca75d83c7dfe00af392f4768563c317e4d24e9de69ed8b1 WatchSource:0}: Error finding container 7a7eafabaca1fc123ca75d83c7dfe00af392f4768563c317e4d24e9de69ed8b1: Status 404 returned error can't find the container with id 7a7eafabaca1fc123ca75d83c7dfe00af392f4768563c317e4d24e9de69ed8b1 Jul 28 14:20:00.121984 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.121907 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cbdbcdff6-gvqcz"] Jul 28 14:20:00.124643 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.124616 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8"] Jul 28 14:20:00.125861 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:00.125831 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917ce53a_a02b_4fa9_9c8d_37861fdba354.slice/crio-b22b78d7ba1791fd0dbe0a2c5aea4642aa888d17327dd66def975ec6d296fe64 WatchSource:0}: Error finding container b22b78d7ba1791fd0dbe0a2c5aea4642aa888d17327dd66def975ec6d296fe64: Status 404 returned error can't find the container with id b22b78d7ba1791fd0dbe0a2c5aea4642aa888d17327dd66def975ec6d296fe64 Jul 28 14:20:00.127004 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:00.126981 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b9f21d_0e5d_48fd_87b1_f8d66f294106.slice/crio-06ea6241bc29ad00cc371a659c8fea550a73930f4af2f54aa1faf77ab5074f05 WatchSource:0}: Error finding container 06ea6241bc29ad00cc371a659c8fea550a73930f4af2f54aa1faf77ab5074f05: Status 404 returned error can't find the container with id 06ea6241bc29ad00cc371a659c8fea550a73930f4af2f54aa1faf77ab5074f05 Jul 28 14:20:00.243049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.243015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" event={"ID":"a2b9f21d-0e5d-48fd-87b1-f8d66f294106","Type":"ContainerStarted","Data":"a29cecda6e4a8eaa832840a8287429f5924f2f6f027507864078fc53f47a4640"} Jul 28 14:20:00.243415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.243072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" event={"ID":"a2b9f21d-0e5d-48fd-87b1-f8d66f294106","Type":"ContainerStarted","Data":"06ea6241bc29ad00cc371a659c8fea550a73930f4af2f54aa1faf77ab5074f05"} Jul 28 14:20:00.244705 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.244632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" event={"ID":"4b586543-0933-4f68-8be0-fce454e73554","Type":"ContainerStarted","Data":"fc0586e3d1eeab39926843c7a724271bd91b40b5a9938867e7e3fce581d894a4"} Jul 28 14:20:00.246701 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.246405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" event={"ID":"917ce53a-a02b-4fa9-9c8d-37861fdba354","Type":"ContainerStarted","Data":"08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4"} Jul 28 14:20:00.246701 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.246454 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:20:00.246701 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.246474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" event={"ID":"917ce53a-a02b-4fa9-9c8d-37861fdba354","Type":"ContainerStarted","Data":"b22b78d7ba1791fd0dbe0a2c5aea4642aa888d17327dd66def975ec6d296fe64"} Jul 28 14:20:00.248087 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.248033 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qlhl" event={"ID":"6f183322-bfbb-4763-b013-72e012cfa318","Type":"ContainerStarted","Data":"9418b9d643580f60474bc1efa0ffe4cd6fbb7957ddbea7e1692576dbb54bbbcb"} Jul 28 14:20:00.249481 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.249354 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" event={"ID":"8c1a2e61-3056-4e8b-8118-d17992749b0a","Type":"ContainerStarted","Data":"7a7eafabaca1fc123ca75d83c7dfe00af392f4768563c317e4d24e9de69ed8b1"} Jul 28 14:20:00.269367 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.269331 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kp7k8" podStartSLOduration=1.269315099 podStartE2EDuration="1.269315099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:00.268422866 +0000 UTC m=+101.933664688" watchObservedRunningTime="2026-07-28 14:20:00.269315099 +0000 UTC m=+101.934556922" Jul 28 14:20:00.293956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:00.293913 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" podStartSLOduration=101.293887416 podStartE2EDuration="1m41.293887416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:00.29377523 +0000 UTC m=+101.959017046" watchObservedRunningTime="2026-07-28 14:20:00.293887416 +0000 UTC m=+101.959129245" Jul 28 14:20:01.165959 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:01.165392 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fpqnz" Jul 28 14:20:04.052851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.052802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:04.091442 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.091405 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mlmrc"] Jul 28 14:20:04.133884 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.133859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mlmrc"] Jul 28 14:20:04.134046 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.133969 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.136743 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.136720 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 14:20:04.136878 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.136756 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-94998\"" Jul 28 14:20:04.136878 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.136832 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 14:20:04.137029 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.136720 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 14:20:04.155288 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.155258 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:04.253978 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.253943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:04.270142 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.270116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b61d58bd-e22c-4535-b49d-96a879e8b042-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.270262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.270235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhzn\" (UniqueName: \"kubernetes.io/projected/b61d58bd-e22c-4535-b49d-96a879e8b042-kube-api-access-jmhzn\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.270314 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.270260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61d58bd-e22c-4535-b49d-96a879e8b042-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.270314 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.270292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b61d58bd-e22c-4535-b49d-96a879e8b042-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.371020 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.370958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhzn\" (UniqueName: \"kubernetes.io/projected/b61d58bd-e22c-4535-b49d-96a879e8b042-kube-api-access-jmhzn\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.371020 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.370999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61d58bd-e22c-4535-b49d-96a879e8b042-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.371205 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.371035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b61d58bd-e22c-4535-b49d-96a879e8b042-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.371205 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.371084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b61d58bd-e22c-4535-b49d-96a879e8b042-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.371770 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.371744 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b61d58bd-e22c-4535-b49d-96a879e8b042-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.373662 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.373633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b61d58bd-e22c-4535-b49d-96a879e8b042-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.373931 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.373903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b61d58bd-e22c-4535-b49d-96a879e8b042-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.379015 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.378977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhzn\" (UniqueName: \"kubernetes.io/projected/b61d58bd-e22c-4535-b49d-96a879e8b042-kube-api-access-jmhzn\") pod \"prometheus-operator-84d69f794d-mlmrc\" (UID: \"b61d58bd-e22c-4535-b49d-96a879e8b042\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:04.446098 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:04.446076 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" Jul 28 14:20:07.608532 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:07.608214 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mlmrc"] Jul 28 14:20:07.613010 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:07.612980 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61d58bd_e22c_4535_b49d_96a879e8b042.slice/crio-e696a81e9826449a9963162188cd95d589fbea2f9e05e97b3028c0d155be9f7d WatchSource:0}: Error finding container e696a81e9826449a9963162188cd95d589fbea2f9e05e97b3028c0d155be9f7d: Status 404 returned error can't find the container with id e696a81e9826449a9963162188cd95d589fbea2f9e05e97b3028c0d155be9f7d Jul 28 14:20:07.667229 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:07.667178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:08.290859 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.290731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qlhl" event={"ID":"6f183322-bfbb-4763-b013-72e012cfa318","Type":"ContainerStarted","Data":"6b27226f4d6b135fd3db18e40c3ee0e8c7a4461245cc30d2fc9bba1d4467b024"} Jul 28 14:20:08.290859 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.290800 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7qlhl" event={"ID":"6f183322-bfbb-4763-b013-72e012cfa318","Type":"ContainerStarted","Data":"f833ddef21bd77b118e434c0413ae266e95458c092048731e201b2c32de86a3c"} Jul 28 14:20:08.291147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.290897 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7qlhl" Jul 28 14:20:08.292622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.292593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" event={"ID":"8c1a2e61-3056-4e8b-8118-d17992749b0a","Type":"ContainerStarted","Data":"a2892382e4b3c7e48666182e528e82d4d002f0e4f6dc2ef7d6e26c0f013ccd27"} Jul 28 14:20:08.292779 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.292763 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:20:08.294614 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.294571 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" Jul 28 14:20:08.295396 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.295342 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" event={"ID":"b61d58bd-e22c-4535-b49d-96a879e8b042","Type":"ContainerStarted","Data":"e696a81e9826449a9963162188cd95d589fbea2f9e05e97b3028c0d155be9f7d"} Jul 28 14:20:08.297534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.297510 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm8mr" event={"ID":"6aec9c3a-0460-4712-b808-9665f7eb3d39","Type":"ContainerStarted","Data":"c2c324d99edf4f72701e3f944efd3dec1c4e681b56295a0ed66d027cc564cb87"} Jul 28 14:20:08.299265 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.299237 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" event={"ID":"4b586543-0933-4f68-8be0-fce454e73554","Type":"ContainerStarted","Data":"354abf5e4b313c47c3079f77bfe22a9710a5e1ee1b09f366ddc9c5ca3e02533b"} Jul 28 14:20:08.312211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.312160 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7qlhl" podStartSLOduration=65.683442955 podStartE2EDuration="1m13.312142745s" totalImagesPullingTime="7.62869979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:08.310129496 +0000 UTC m=+109.975371321" watchObservedRunningTime="2026-07-28 14:20:08.312142745 +0000 UTC m=+109.977384569" Jul 28 14:20:08.331224 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.331132 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-p9h77" podStartSLOduration=2.231952106 podStartE2EDuration="9.331114573s" totalImagesPullingTime="7.099162467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:08.330523825 +0000 UTC m=+109.995765649" watchObservedRunningTime="2026-07-28 14:20:08.331114573 +0000 UTC m=+109.996356397" Jul 28 14:20:08.347071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.347028 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f8944f5dd-cchnh" podStartSLOduration=1.924959895 podStartE2EDuration="9.347010517s" totalImagesPullingTime="7.422050622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:08.346865731 +0000 UTC m=+110.012107556" watchObservedRunningTime="2026-07-28 14:20:08.347010517 +0000 UTC m=+110.012252341" Jul 28 14:20:08.441296 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.441234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:08.543072 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:08.542992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:09.264787 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:09.264707 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:09.307034 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:09.306993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" event={"ID":"b61d58bd-e22c-4535-b49d-96a879e8b042","Type":"ContainerStarted","Data":"a9edc152e9fcdd6fcb9459d7694ff4b0560d9d8f8ca497083a5e91a9379541be"} Jul 28 14:20:09.307190 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:09.307097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" event={"ID":"b61d58bd-e22c-4535-b49d-96a879e8b042","Type":"ContainerStarted","Data":"22ca6fbef956920bf713d5c491bf52be31565569f845f47b64e6136a0b88fd87"} Jul 28 14:20:09.325401 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:09.325349 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-mlmrc" podStartSLOduration=3.985222293 podStartE2EDuration="5.325329937s" totalImagesPullingTime="1.340107644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:09.324790685 +0000 UTC m=+110.990032509" watchObservedRunningTime="2026-07-28 14:20:09.325329937 +0000 UTC m=+110.990571761" Jul 28 14:20:10.459216 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.459184 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-684567f7fd-mqrfv"] Jul 28 14:20:10.476429 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.476390 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684567f7fd-mqrfv"] Jul 28 14:20:10.476566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.476527 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.479473 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.479246 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 14:20:10.479473 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.479257 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 14:20:10.479473 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.479416 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 14:20:10.479473 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.479432 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 14:20:10.480147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.480067 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-whd2d\"" Jul 28 14:20:10.480147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.480081 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 14:20:10.480147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.480139 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 14:20:10.480405 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.480163 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 14:20:10.485974 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.485458 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 14:20:10.621645 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-service-ca\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.621798 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-oauth-serving-cert\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.621798 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6km\" (UniqueName: \"kubernetes.io/projected/edf20565-d89e-4155-8527-f335fae3c04b-kube-api-access-bn6km\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.621905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-console-config\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.621905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621834 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-serving-cert\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.621905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-trusted-ca-bundle\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.622014 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.621946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-oauth-config\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.667938 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.667893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:10.722712 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.722629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-trusted-ca-bundle\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.722712 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.722704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-oauth-config\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.722892 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.722733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-service-ca\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.722945 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.722892 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-oauth-serving-cert\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.723004 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.722949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6km\" (UniqueName: \"kubernetes.io/projected/edf20565-d89e-4155-8527-f335fae3c04b-kube-api-access-bn6km\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.723061 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.723014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-console-config\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.723189 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.723130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-serving-cert\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.723464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.723443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-service-ca\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.723951 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.723928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-oauth-serving-cert\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.724041 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.723928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-console-config\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.724119 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.724061 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-trusted-ca-bundle\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.725252 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.725231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-oauth-config\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.725875 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.725851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-serving-cert\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.731861 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.731835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6km\" (UniqueName: \"kubernetes.io/projected/edf20565-d89e-4155-8527-f335fae3c04b-kube-api-access-bn6km\") pod \"console-684567f7fd-mqrfv\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:10.769206 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.769180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:10.789217 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:10.789186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:11.213791 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.213716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:11.468622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.468537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:11.517816 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.517791 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-btszk"] Jul 28 14:20:11.521184 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.521163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.524400 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.524381 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 14:20:11.527104 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.527083 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 14:20:11.527215 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.527192 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zplpq\"" Jul 28 14:20:11.527427 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.527408 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 14:20:11.544030 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.544007 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-btszk"] Jul 28 14:20:11.546256 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.546236 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ftxr8"] Jul 28 14:20:11.549487 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.549470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.552475 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.552425 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k2gn6\"" Jul 28 14:20:11.552475 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.552469 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 14:20:11.552642 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.552475 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 14:20:11.552642 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.552483 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 14:20:11.596336 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.596303 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:11.630395 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.630358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db4b61fd-73ee-4028-9f67-d291563e3e28-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.630510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.630422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.630569 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.630500 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db4b61fd-73ee-4028-9f67-d291563e3e28-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.630634 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.630603 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.630708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.630634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzn6\" (UniqueName: \"kubernetes.io/projected/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-api-access-rfzn6\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.630708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.630693 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.683902 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.683832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:11.724535 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.724514 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-684567f7fd-mqrfv"] Jul 28 14:20:11.727558 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:11.727537 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf20565_d89e_4155_8527_f335fae3c04b.slice/crio-224dc5c08369effcb63797101960a695433478af73b198b5ecd3ef2a4a0ad311 WatchSource:0}: Error finding container 224dc5c08369effcb63797101960a695433478af73b198b5ecd3ef2a4a0ad311: Status 404 returned error can't find the container with id 224dc5c08369effcb63797101960a695433478af73b198b5ecd3ef2a4a0ad311 Jul 28 14:20:11.731695 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.731570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db4b61fd-73ee-4028-9f67-d291563e3e28-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.731910 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.731876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db4b61fd-73ee-4028-9f67-d291563e3e28-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-textfile\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-sys\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732175 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db4b61fd-73ee-4028-9f67-d291563e3e28-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-wtmp\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732228 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a4434f1-2784-4ba2-9f64-0742cc2429db-metrics-client-ca\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-tls\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732430 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-root\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzhc\" (UniqueName: \"kubernetes.io/projected/2a4434f1-2784-4ba2-9f64-0742cc2429db-kube-api-access-4jzhc\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzn6\" (UniqueName: \"kubernetes.io/projected/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-api-access-rfzn6\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.732782 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:20:11.732739 2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 28 14:20:11.733261 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:20:11.732851 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-tls podName:db4b61fd-73ee-4028-9f67-d291563e3e28 nodeName:}" failed. No retries permitted until 2026-07-28 14:20:12.232794439 +0000 UTC m=+113.898036254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-btszk" (UID: "db4b61fd-73ee-4028-9f67-d291563e3e28") : secret "kube-state-metrics-tls" not found Jul 28 14:20:11.733261 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.732899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db4b61fd-73ee-4028-9f67-d291563e3e28-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.735042 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.735016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.746409 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.746387 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzn6\" (UniqueName: \"kubernetes.io/projected/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-api-access-rfzn6\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:11.785300 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.785277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:11.833924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.833902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a4434f1-2784-4ba2-9f64-0742cc2429db-metrics-client-ca\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.833924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.833925 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-tls\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834035 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.833942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-root\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834035 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.833960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834035 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834025 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-root\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834174 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzhc\" (UniqueName: \"kubernetes.io/projected/2a4434f1-2784-4ba2-9f64-0742cc2429db-kube-api-access-4jzhc\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834298 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-textfile\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834311 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-sys\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-wtmp\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a4434f1-2784-4ba2-9f64-0742cc2429db-metrics-client-ca\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-sys\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-textfile\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834778 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834611 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-wtmp\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.834835 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.834782 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-accelerators-collector-config\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.836267 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.836249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.836408 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.836393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a4434f1-2784-4ba2-9f64-0742cc2429db-node-exporter-tls\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.841835 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.841813 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzhc\" (UniqueName: \"kubernetes.io/projected/2a4434f1-2784-4ba2-9f64-0742cc2429db-kube-api-access-4jzhc\") pod \"node-exporter-ftxr8\" (UID: \"2a4434f1-2784-4ba2-9f64-0742cc2429db\") " pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.859395 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.859379 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ftxr8" Jul 28 14:20:11.869738 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:11.869713 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4434f1_2784_4ba2_9f64_0742cc2429db.slice/crio-e80628fafc940e126290fdbbc049c5f92c93789bdc09c5f089ecd2c2a2bb586c WatchSource:0}: Error finding container e80628fafc940e126290fdbbc049c5f92c93789bdc09c5f089ecd2c2a2bb586c: Status 404 returned error can't find the container with id e80628fafc940e126290fdbbc049c5f92c93789bdc09c5f089ecd2c2a2bb586c Jul 28 14:20:11.889258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.889234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:11.980956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:11.980888 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:12.050705 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.050669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:12.148008 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.147963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:12.238537 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.238469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:12.241273 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.241246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db4b61fd-73ee-4028-9f67-d291563e3e28-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-btszk\" (UID: \"db4b61fd-73ee-4028-9f67-d291563e3e28\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:12.321916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.321825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftxr8" event={"ID":"2a4434f1-2784-4ba2-9f64-0742cc2429db","Type":"ContainerStarted","Data":"e80628fafc940e126290fdbbc049c5f92c93789bdc09c5f089ecd2c2a2bb586c"} Jul 28 14:20:12.323252 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.323182 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684567f7fd-mqrfv" event={"ID":"edf20565-d89e-4155-8527-f335fae3c04b","Type":"ContainerStarted","Data":"224dc5c08369effcb63797101960a695433478af73b198b5ecd3ef2a4a0ad311"} Jul 28 14:20:12.326004 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.325943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm8mr" event={"ID":"6aec9c3a-0460-4712-b808-9665f7eb3d39","Type":"ContainerStarted","Data":"af7e37c7db512f27d283d70d4a9da5c103d23a25e5d90db12b8e0b956e8d2fb3"} Jul 28 14:20:12.360056 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.360030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:12.432446 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.431544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" Jul 28 14:20:12.490175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.489926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:12.574228 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.574200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:12.645173 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:12.644775 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-btszk"] Jul 28 14:20:13.006806 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:13.006762 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4b61fd_73ee_4028_9f67_d291563e3e28.slice/crio-c460b3c1697818ad8838fa64532bc9cb3cf1cf3a65bb4341e2ce533fe3267b51 WatchSource:0}: Error finding container c460b3c1697818ad8838fa64532bc9cb3cf1cf3a65bb4341e2ce533fe3267b51: Status 404 returned error can't find the container with id c460b3c1697818ad8838fa64532bc9cb3cf1cf3a65bb4341e2ce533fe3267b51 Jul 28 14:20:13.333330 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:13.333293 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" event={"ID":"db4b61fd-73ee-4028-9f67-d291563e3e28","Type":"ContainerStarted","Data":"c460b3c1697818ad8838fa64532bc9cb3cf1cf3a65bb4341e2ce533fe3267b51"} Jul 28 14:20:13.335861 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:13.335733 2562 generic.go:356] "Generic (PLEG): container finished" podID="2a4434f1-2784-4ba2-9f64-0742cc2429db" containerID="704f76ec44aff8675640a29af0fecdb55c6f77ce5ab3b84894c5e9fee99d6c99" exitCode=0 Jul 28 14:20:13.335861 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:13.335804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftxr8" event={"ID":"2a4434f1-2784-4ba2-9f64-0742cc2429db","Type":"ContainerDied","Data":"704f76ec44aff8675640a29af0fecdb55c6f77ce5ab3b84894c5e9fee99d6c99"} Jul 28 14:20:13.542432 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:13.542157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:14.342216 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.342180 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftxr8" event={"ID":"2a4434f1-2784-4ba2-9f64-0742cc2429db","Type":"ContainerStarted","Data":"9c827854d85828da9f1328812028ac8cda6cb4462873b7af56301230cf9a1179"} Jul 28 14:20:14.342370 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.342241 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ftxr8" event={"ID":"2a4434f1-2784-4ba2-9f64-0742cc2429db","Type":"ContainerStarted","Data":"052a3a8351bb8283e9743c20314dbeb1a4140b3fa74d2ac504395401dbed02bf"} Jul 28 14:20:14.370012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.369957 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ftxr8" podStartSLOduration=2.080762296 podStartE2EDuration="3.369934465s" totalImagesPullingTime="1.1878128s" totalInitContainerRuntime="101.359369ms" isStatefulPod=true podCreationTimestamp="2026-07-28 14:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:14.369178163 +0000 UTC m=+116.034419985" watchObservedRunningTime="2026-07-28 14:20:14.369934465 +0000 UTC m=+116.035176290" Jul 28 14:20:14.430309 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.430273 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:14.517269 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.517224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:14.730391 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.730290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:14.829605 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:14.829567 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:16.147115 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.147083 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:16.308045 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.308001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:16.353855 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.353828 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm"] Jul 28 14:20:16.358129 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.358110 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.365653 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.365632 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 14:20:16.365833 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.365656 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qr5qw\"" Jul 28 14:20:16.376747 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.376714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/555011e8-077d-4f1b-a46f-06212830bef4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.376883 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.376768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrq8\" (UniqueName: \"kubernetes.io/projected/555011e8-077d-4f1b-a46f-06212830bef4-kube-api-access-dbrq8\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.377009 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.376980 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm"] Jul 28 14:20:16.419987 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.419915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:16.477893 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.477867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/555011e8-077d-4f1b-a46f-06212830bef4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.478031 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.477910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrq8\" (UniqueName: \"kubernetes.io/projected/555011e8-077d-4f1b-a46f-06212830bef4-kube-api-access-dbrq8\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.478096 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:20:16.478032 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 28 14:20:16.478149 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:20:16.478109 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555011e8-077d-4f1b-a46f-06212830bef4-monitoring-plugin-cert podName:555011e8-077d-4f1b-a46f-06212830bef4 nodeName:}" failed. No retries permitted until 2026-07-28 14:20:16.978088032 +0000 UTC m=+118.643329848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/555011e8-077d-4f1b-a46f-06212830bef4-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-p26sm" (UID: "555011e8-077d-4f1b-a46f-06212830bef4") : secret "monitoring-plugin-cert" not found Jul 28 14:20:16.505192 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.505164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrq8\" (UniqueName: \"kubernetes.io/projected/555011e8-077d-4f1b-a46f-06212830bef4-kube-api-access-dbrq8\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.598112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.598078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:16.983392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.983360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/555011e8-077d-4f1b-a46f-06212830bef4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:16.986119 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:16.986090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/555011e8-077d-4f1b-a46f-06212830bef4-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-p26sm\" (UID: \"555011e8-077d-4f1b-a46f-06212830bef4\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:17.269256 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:17.269236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:17.357127 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:17.357049 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" event={"ID":"db4b61fd-73ee-4028-9f67-d291563e3e28","Type":"ContainerStarted","Data":"9025f6c426ecb4dc54b3b285285b6cca78758400a1a109385a7896e844d11cec"} Jul 28 14:20:17.361139 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:17.361110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bm8mr" event={"ID":"6aec9c3a-0460-4712-b808-9665f7eb3d39","Type":"ContainerStarted","Data":"efede6c6f58fcb1686ec667435516ca4977bd9ae317921cd975f6977d899a30a"} Jul 28 14:20:17.382934 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:17.382848 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bm8mr" podStartSLOduration=33.163946574 podStartE2EDuration="51.382827461s" totalImagesPullingTime="18.218880887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:19:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:17.38143212 +0000 UTC m=+119.046673947" watchObservedRunningTime="2026-07-28 14:20:17.382827461 +0000 UTC m=+119.048069286" Jul 28 14:20:17.453196 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:17.453128 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm"] Jul 28 14:20:17.458985 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:17.458943 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555011e8_077d_4f1b_a46f_06212830bef4.slice/crio-652b80bc8721879e744746b7aa14032826bcfaa9cb2db092c91c7d0eae8888f4 WatchSource:0}: Error finding container 652b80bc8721879e744746b7aa14032826bcfaa9cb2db092c91c7d0eae8888f4: Status 404 returned error can't find the container with id 652b80bc8721879e744746b7aa14032826bcfaa9cb2db092c91c7d0eae8888f4 Jul 28 14:20:17.765645 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:17.765608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:18.308890 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.308861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7qlhl" Jul 28 14:20:18.368338 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.368300 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" event={"ID":"db4b61fd-73ee-4028-9f67-d291563e3e28","Type":"ContainerStarted","Data":"a19c2e17462be9b3a125ef5e1760ef32e3f8ccf7dd6cca332961b5f93583cd03"} Jul 28 14:20:18.368505 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.368370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" event={"ID":"db4b61fd-73ee-4028-9f67-d291563e3e28","Type":"ContainerStarted","Data":"5bd486b6b535420f53569744601f8bc28f118384aed6c80a0aa4e7a065a20215"} Jul 28 14:20:18.370347 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.370308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684567f7fd-mqrfv" event={"ID":"edf20565-d89e-4155-8527-f335fae3c04b","Type":"ContainerStarted","Data":"0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371"} Jul 28 14:20:18.371812 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.371782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" event={"ID":"555011e8-077d-4f1b-a46f-06212830bef4","Type":"ContainerStarted","Data":"652b80bc8721879e744746b7aa14032826bcfaa9cb2db092c91c7d0eae8888f4"} Jul 28 14:20:18.388063 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.388004 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-btszk" podStartSLOduration=3.116793464 podStartE2EDuration="7.38798486s" totalImagesPullingTime="4.271191396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:18.386170566 +0000 UTC m=+120.051412390" watchObservedRunningTime="2026-07-28 14:20:18.38798486 +0000 UTC m=+120.053226685" Jul 28 14:20:18.404692 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:18.404635 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-684567f7fd-mqrfv" podStartSLOduration=2.844005117 podStartE2EDuration="8.404622587s" totalImagesPullingTime="5.56061747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:18.403903216 +0000 UTC m=+120.069145041" watchObservedRunningTime="2026-07-28 14:20:18.404622587 +0000 UTC m=+120.069864410" Jul 28 14:20:19.378942 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:19.378902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" event={"ID":"555011e8-077d-4f1b-a46f-06212830bef4","Type":"ContainerStarted","Data":"d357bedfd0284be3abd02db1b73071ecbda5c0a6aab46d523a807c801edf8f8d"} Jul 28 14:20:19.379390 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:19.378971 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:19.383858 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:19.383832 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" Jul 28 14:20:19.395909 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:19.395861 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-p26sm" podStartSLOduration=1.866045117 podStartE2EDuration="3.395845723s" totalImagesPullingTime="1.529800606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:19.395026596 +0000 UTC m=+121.060268422" watchObservedRunningTime="2026-07-28 14:20:19.395845723 +0000 UTC m=+121.061087538" Jul 28 14:20:19.932476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:19.932443 2562 patch_prober.go:28] interesting pod/image-registry-5cbdbcdff6-gvqcz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 28 14:20:19.932616 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:19.932491 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" podUID="917ce53a-a02b-4fa9-9c8d-37861fdba354" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 14:20:20.789797 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:20.789767 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:20.790170 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:20.789843 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:20.794183 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:20.794161 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:20.819308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:20.819285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:20.913947 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:20.913918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:21.263086 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:21.263063 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:20:21.389744 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:21.389722 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:21.459138 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:21.459107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:21.559997 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:21.559962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:21.659619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:21.659593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:21.970314 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:21.970241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:22.096374 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:22.096340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:22.269714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:22.269628 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cbdbcdff6-gvqcz"] Jul 28 14:20:22.421127 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:22.421096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:23.630914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:23.630880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:23.710891 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:23.710862 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:26.180515 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:26.180472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:26.246919 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:26.246882 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:27.955692 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:27.955643 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684567f7fd-mqrfv"] Jul 28 14:20:28.071698 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.071660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:28.171001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.170970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:28.272716 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.272641 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:28.678440 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.678414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:20:28.680828 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.680793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0fded26-1385-4fb1-93df-dea08d2b474c-metrics-certs\") pod \"network-metrics-daemon-8sjnc\" (UID: \"f0fded26-1385-4fb1-93df-dea08d2b474c\") " pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:20:28.695927 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.695907 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4gd6j\"" Jul 28 14:20:28.703788 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.703771 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8sjnc" Jul 28 14:20:28.833205 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:28.833170 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8sjnc"] Jul 28 14:20:28.836799 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:28.836775 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fded26_1385_4fb1_93df_dea08d2b474c.slice/crio-29f62f21ce705e9fb8a63f8fdcdb5557d730ec1776f8557aa8817a47364f6625 WatchSource:0}: Error finding container 29f62f21ce705e9fb8a63f8fdcdb5557d730ec1776f8557aa8817a47364f6625: Status 404 returned error can't find the container with id 29f62f21ce705e9fb8a63f8fdcdb5557d730ec1776f8557aa8817a47364f6625 Jul 28 14:20:29.418267 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:29.418236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8sjnc" event={"ID":"f0fded26-1385-4fb1-93df-dea08d2b474c","Type":"ContainerStarted","Data":"29f62f21ce705e9fb8a63f8fdcdb5557d730ec1776f8557aa8817a47364f6625"} Jul 28 14:20:30.423714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:30.423661 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8sjnc" event={"ID":"f0fded26-1385-4fb1-93df-dea08d2b474c","Type":"ContainerStarted","Data":"025e1e6cc739d5c98f73bd35e94bfcfd20883861b921c4fed2136cfca8b2783a"} Jul 28 14:20:30.424086 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:30.423739 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8sjnc" event={"ID":"f0fded26-1385-4fb1-93df-dea08d2b474c","Type":"ContainerStarted","Data":"293926fbdfce5ff2a0bfbd8274a37733f50c5af505623544ce2fcdf7a5653d32"} Jul 28 14:20:30.448729 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:30.448667 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8sjnc" podStartSLOduration=130.527264687 podStartE2EDuration="2m11.44865235s" totalImagesPullingTime="921.387663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:30.447710609 +0000 UTC m=+132.112952431" watchObservedRunningTime="2026-07-28 14:20:30.44865235 +0000 UTC m=+132.113894182" Jul 28 14:20:30.617408 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:30.617379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:30.710967 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:30.710903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:30.800037 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:30.800011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:31.943637 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:31.943593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:32.060965 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:32.060933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:33.208008 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.207973 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556d7b4bf4-knb4c"] Jul 28 14:20:33.213600 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.213581 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.240729 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.240709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556d7b4bf4-knb4c"] Jul 28 14:20:33.313510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.313478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-oauth-config\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.313725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.313705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-config\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.313836 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.313824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvhxm\" (UniqueName: \"kubernetes.io/projected/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-kube-api-access-jvhxm\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.313963 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.313951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-service-ca\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.314057 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.314046 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-serving-cert\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.314147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.314136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-oauth-serving-cert\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.314242 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.314225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-trusted-ca-bundle\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.365135 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.365109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:33.415565 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-service-ca\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.415670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-serving-cert\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.415670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-oauth-serving-cert\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.415670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-trusted-ca-bundle\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.415670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-oauth-config\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.415889 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-config\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.415889 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.415734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhxm\" (UniqueName: \"kubernetes.io/projected/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-kube-api-access-jvhxm\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.416367 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.416338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-service-ca\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.416367 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.416351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-oauth-serving-cert\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.416510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.416455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-config\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.416725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.416697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-trusted-ca-bundle\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.418555 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.418528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-oauth-config\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.418695 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.418655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-serving-cert\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.428821 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.428800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhxm\" (UniqueName: \"kubernetes.io/projected/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-kube-api-access-jvhxm\") pod \"console-556d7b4bf4-knb4c\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.484838 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.484780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:33.522413 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.522390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:33.649612 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:33.649437 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556d7b4bf4-knb4c"] Jul 28 14:20:33.652139 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:20:33.652110 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14eb7585_632d_4e3b_ad3d_2278c3fabcd0.slice/crio-4861443f7eb425c22df651e2fd12ab8b0e0e39d2ced84dafebafa207f72de465 WatchSource:0}: Error finding container 4861443f7eb425c22df651e2fd12ab8b0e0e39d2ced84dafebafa207f72de465: Status 404 returned error can't find the container with id 4861443f7eb425c22df651e2fd12ab8b0e0e39d2ced84dafebafa207f72de465 Jul 28 14:20:34.118348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:34.118312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:34.206629 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:34.206573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:34.441540 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:34.441480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556d7b4bf4-knb4c" event={"ID":"14eb7585-632d-4e3b-ad3d-2278c3fabcd0","Type":"ContainerStarted","Data":"f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a"} Jul 28 14:20:34.441540 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:34.441528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556d7b4bf4-knb4c" event={"ID":"14eb7585-632d-4e3b-ad3d-2278c3fabcd0","Type":"ContainerStarted","Data":"4861443f7eb425c22df651e2fd12ab8b0e0e39d2ced84dafebafa207f72de465"} Jul 28 14:20:34.467805 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:34.467761 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556d7b4bf4-knb4c" podStartSLOduration=1.467744487 podStartE2EDuration="1.467744487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:20:34.467360398 +0000 UTC m=+136.132602221" watchObservedRunningTime="2026-07-28 14:20:34.467744487 +0000 UTC m=+136.132986310" Jul 28 14:20:35.313216 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:35.313184 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:35.396555 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:35.396520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:36.452634 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:36.452599 2562 generic.go:356] "Generic (PLEG): container finished" podID="e16a89f6-30f6-4e0d-9823-df204fb02fdd" containerID="47e572d63fc39a3fe0998171b50357d8c18d3c0cc13fb2e92eb1f0d2baaaffeb" exitCode=0 Jul 28 14:20:36.453107 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:36.452699 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qml4d" event={"ID":"e16a89f6-30f6-4e0d-9823-df204fb02fdd","Type":"ContainerDied","Data":"47e572d63fc39a3fe0998171b50357d8c18d3c0cc13fb2e92eb1f0d2baaaffeb"} Jul 28 14:20:36.453107 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:36.453048 2562 scope.go:117] "RemoveContainer" containerID="47e572d63fc39a3fe0998171b50357d8c18d3c0cc13fb2e92eb1f0d2baaaffeb" Jul 28 14:20:37.457932 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:37.457902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qml4d" event={"ID":"e16a89f6-30f6-4e0d-9823-df204fb02fdd","Type":"ContainerStarted","Data":"c07e86425fa36589a22cde206a73a62eac9a7656aa02c1c815dace9099509975"} Jul 28 14:20:43.212565 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.212515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:43.313291 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.313252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:43.413877 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.413854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:43.512899 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.512839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:43.522652 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.522636 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:43.522742 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.522716 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:43.527262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:43.527245 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:44.486728 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:44.486701 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:20:44.668248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:44.668221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:44.769444 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:44.769388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:44.835748 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:44.835721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:47.288737 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:47.288667 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" podUID="917ce53a-a02b-4fa9-9c8d-37861fdba354" containerName="registry" containerID="cri-o://08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4" gracePeriod=30 Jul 28 14:20:48.062015 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.061990 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:20:48.120321 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120272 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-image-registry-private-configuration\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120341 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-trusted-ca\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120392 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-certificates\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120472 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-bound-sa-token\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-installation-pull-secrets\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120583 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120769 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120658 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/917ce53a-a02b-4fa9-9c8d-37861fdba354-ca-trust-extracted\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.120769 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.120721 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvk9p\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-kube-api-access-pvk9p\") pod \"917ce53a-a02b-4fa9-9c8d-37861fdba354\" (UID: \"917ce53a-a02b-4fa9-9c8d-37861fdba354\") " Jul 28 14:20:48.122162 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.122112 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:20:48.122162 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.122128 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:20:48.124073 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.124046 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-kube-api-access-pvk9p" (OuterVolumeSpecName: "kube-api-access-pvk9p") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "kube-api-access-pvk9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:20:48.124283 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.124259 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:20:48.124370 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.124326 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:20:48.124490 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.124470 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:20:48.124640 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.124617 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:20:48.132662 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.132629 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917ce53a-a02b-4fa9-9c8d-37861fdba354-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "917ce53a-a02b-4fa9-9c8d-37861fdba354" (UID: "917ce53a-a02b-4fa9-9c8d-37861fdba354"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:20:48.221793 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221745 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-certificates\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221793 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221767 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-bound-sa-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221793 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221778 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-installation-pull-secrets\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221793 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221787 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-registry-tls\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221793 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221795 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/917ce53a-a02b-4fa9-9c8d-37861fdba354-ca-trust-extracted\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221803 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvk9p\" (UniqueName: \"kubernetes.io/projected/917ce53a-a02b-4fa9-9c8d-37861fdba354-kube-api-access-pvk9p\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221811 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/917ce53a-a02b-4fa9-9c8d-37861fdba354-image-registry-private-configuration\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.221991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.221821 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/917ce53a-a02b-4fa9-9c8d-37861fdba354-trusted-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:48.499052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.498984 2562 generic.go:356] "Generic (PLEG): container finished" podID="917ce53a-a02b-4fa9-9c8d-37861fdba354" containerID="08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4" exitCode=0 Jul 28 14:20:48.499052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.499044 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" Jul 28 14:20:48.499539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.499074 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" event={"ID":"917ce53a-a02b-4fa9-9c8d-37861fdba354","Type":"ContainerDied","Data":"08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4"} Jul 28 14:20:48.499539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.499130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cbdbcdff6-gvqcz" event={"ID":"917ce53a-a02b-4fa9-9c8d-37861fdba354","Type":"ContainerDied","Data":"b22b78d7ba1791fd0dbe0a2c5aea4642aa888d17327dd66def975ec6d296fe64"} Jul 28 14:20:48.499539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.499152 2562 scope.go:117] "RemoveContainer" containerID="08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4" Jul 28 14:20:48.508426 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.508408 2562 scope.go:117] "RemoveContainer" containerID="08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4" Jul 28 14:20:48.508707 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:20:48.508663 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4\": container with ID starting with 08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4 not found: ID does not exist" containerID="08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4" Jul 28 14:20:48.508776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.508718 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4"} err="failed to get container status \"08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4\": rpc error: code = NotFound desc = could not find container \"08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4\": container with ID starting with 08b05d263746def8b7066b81248ee98574199624475559846b963111fd1991a4 not found: ID does not exist" Jul 28 14:20:48.522483 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.522458 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cbdbcdff6-gvqcz"] Jul 28 14:20:48.529997 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.529977 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cbdbcdff6-gvqcz"] Jul 28 14:20:48.887168 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:48.887135 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="917ce53a-a02b-4fa9-9c8d-37861fdba354" path="/var/lib/kubelet/pods/917ce53a-a02b-4fa9-9c8d-37861fdba354/volumes" Jul 28 14:20:50.284032 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:50.283998 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:50.384765 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:50.384728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:20:52.974564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:52.974519 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-684567f7fd-mqrfv" podUID="edf20565-d89e-4155-8527-f335fae3c04b" containerName="console" containerID="cri-o://0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371" gracePeriod=15 Jul 28 14:20:53.220336 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.220311 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:53.358403 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358380 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-serving-cert\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358508 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358412 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-trusted-ca-bundle\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358508 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358444 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-service-ca\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358508 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358476 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-console-config\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358508 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358502 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-oauth-config\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358667 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358538 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn6km\" (UniqueName: \"kubernetes.io/projected/edf20565-d89e-4155-8527-f335fae3c04b-kube-api-access-bn6km\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358667 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358590 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-oauth-serving-cert\") pod \"edf20565-d89e-4155-8527-f335fae3c04b\" (UID: \"edf20565-d89e-4155-8527-f335fae3c04b\") " Jul 28 14:20:53.358858 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358826 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-service-ca" (OuterVolumeSpecName: "service-ca") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:20:53.358929 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358888 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:20:53.358983 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.358955 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-console-config" (OuterVolumeSpecName: "console-config") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:20:53.359162 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.359132 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:20:53.360731 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.360708 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:20:53.360832 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.360809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf20565-d89e-4155-8527-f335fae3c04b-kube-api-access-bn6km" (OuterVolumeSpecName: "kube-api-access-bn6km") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "kube-api-access-bn6km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:20:53.360871 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.360837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "edf20565-d89e-4155-8527-f335fae3c04b" (UID: "edf20565-d89e-4155-8527-f335fae3c04b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:20:53.459764 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459741 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6km\" (UniqueName: \"kubernetes.io/projected/edf20565-d89e-4155-8527-f335fae3c04b-kube-api-access-bn6km\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.459764 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459761 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-oauth-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.459880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459770 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.459880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459778 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-trusted-ca-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.459880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459788 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.459880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459797 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/edf20565-d89e-4155-8527-f335fae3c04b-console-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.459880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.459804 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/edf20565-d89e-4155-8527-f335fae3c04b-console-oauth-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:20:53.525912 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.525874 2562 generic.go:356] "Generic (PLEG): container finished" podID="edf20565-d89e-4155-8527-f335fae3c04b" containerID="0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371" exitCode=0 Jul 28 14:20:53.526004 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.525941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684567f7fd-mqrfv" event={"ID":"edf20565-d89e-4155-8527-f335fae3c04b","Type":"ContainerDied","Data":"0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371"} Jul 28 14:20:53.526004 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.525951 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-684567f7fd-mqrfv" Jul 28 14:20:53.526071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.526020 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-684567f7fd-mqrfv" event={"ID":"edf20565-d89e-4155-8527-f335fae3c04b","Type":"ContainerDied","Data":"224dc5c08369effcb63797101960a695433478af73b198b5ecd3ef2a4a0ad311"} Jul 28 14:20:53.526071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.526050 2562 scope.go:117] "RemoveContainer" containerID="0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371" Jul 28 14:20:53.535174 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.535157 2562 scope.go:117] "RemoveContainer" containerID="0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371" Jul 28 14:20:53.535456 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:20:53.535436 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371\": container with ID starting with 0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371 not found: ID does not exist" containerID="0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371" Jul 28 14:20:53.535523 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.535468 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371"} err="failed to get container status \"0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371\": rpc error: code = NotFound desc = could not find container \"0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371\": container with ID starting with 0bbc7b6bbeade545e06cc33872c075cdb5527ae3b5a895d06a62c8a61f046371 not found: ID does not exist" Jul 28 14:20:53.550342 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.550317 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-684567f7fd-mqrfv"] Jul 28 14:20:53.553762 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:53.553742 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-684567f7fd-mqrfv"] Jul 28 14:20:54.887373 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:20:54.887335 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf20565-d89e-4155-8527-f335fae3c04b" path="/var/lib/kubelet/pods/edf20565-d89e-4155-8527-f335fae3c04b/volumes" Jul 28 14:21:06.321772 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:06.321730 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:06.421879 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:06.421847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:27.663244 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:27.663207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:27.729936 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:27.729902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:39.606775 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:39.606718 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:39.682719 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:39.682655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:45.044119 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.044086 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69b6f97ccc-nvs87"] Jul 28 14:21:45.044936 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.044908 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf20565-d89e-4155-8527-f335fae3c04b" containerName="console" Jul 28 14:21:45.044936 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.044932 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf20565-d89e-4155-8527-f335fae3c04b" containerName="console" Jul 28 14:21:45.045133 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.044965 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="917ce53a-a02b-4fa9-9c8d-37861fdba354" containerName="registry" Jul 28 14:21:45.045133 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.044973 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="917ce53a-a02b-4fa9-9c8d-37861fdba354" containerName="registry" Jul 28 14:21:45.045133 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.045041 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="917ce53a-a02b-4fa9-9c8d-37861fdba354" containerName="registry" Jul 28 14:21:45.045133 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.045052 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf20565-d89e-4155-8527-f335fae3c04b" containerName="console" Jul 28 14:21:45.048124 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.048103 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.063868 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.063847 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69b6f97ccc-nvs87"] Jul 28 14:21:45.110511 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.110488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:45.119620 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-trusted-ca-bundle\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.119710 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-serving-cert\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.119710 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-console-config\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.119710 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-service-ca\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.119813 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-oauth-serving-cert\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.119813 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-oauth-config\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.119872 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.119808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp6h\" (UniqueName: \"kubernetes.io/projected/e2b70ceb-be47-4209-b150-673ec72fcba1-kube-api-access-pbp6h\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.173121 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.173097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:45.220181 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-console-config\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-service-ca\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-oauth-serving-cert\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220347 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-oauth-config\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220395 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp6h\" (UniqueName: \"kubernetes.io/projected/e2b70ceb-be47-4209-b150-673ec72fcba1-kube-api-access-pbp6h\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220455 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-trusted-ca-bundle\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-serving-cert\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.220962 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.220935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-service-ca\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.221068 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.221053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-console-config\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.221134 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.221070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-oauth-serving-cert\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.221341 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.221318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-trusted-ca-bundle\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.222915 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.222889 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-oauth-config\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.223156 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.223136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-serving-cert\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.229112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.229091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp6h\" (UniqueName: \"kubernetes.io/projected/e2b70ceb-be47-4209-b150-673ec72fcba1-kube-api-access-pbp6h\") pod \"console-69b6f97ccc-nvs87\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.357454 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.357393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:45.488952 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.488923 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69b6f97ccc-nvs87"] Jul 28 14:21:45.492044 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:21:45.492019 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b70ceb_be47_4209_b150_673ec72fcba1.slice/crio-bf91fd546711c6e2650a1e803f6708ca96acc5575b2087f4f3d8efc3baa9b51d WatchSource:0}: Error finding container bf91fd546711c6e2650a1e803f6708ca96acc5575b2087f4f3d8efc3baa9b51d: Status 404 returned error can't find the container with id bf91fd546711c6e2650a1e803f6708ca96acc5575b2087f4f3d8efc3baa9b51d Jul 28 14:21:45.726947 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.726871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b6f97ccc-nvs87" event={"ID":"e2b70ceb-be47-4209-b150-673ec72fcba1","Type":"ContainerStarted","Data":"6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667"} Jul 28 14:21:45.726947 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.726942 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b6f97ccc-nvs87" event={"ID":"e2b70ceb-be47-4209-b150-673ec72fcba1","Type":"ContainerStarted","Data":"bf91fd546711c6e2650a1e803f6708ca96acc5575b2087f4f3d8efc3baa9b51d"} Jul 28 14:21:45.745442 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:45.745394 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69b6f97ccc-nvs87" podStartSLOduration=0.745378706 podStartE2EDuration="745.378706ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:21:45.744285338 +0000 UTC m=+207.409527153" watchObservedRunningTime="2026-07-28 14:21:45.745378706 +0000 UTC m=+207.410620528" Jul 28 14:21:50.307883 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:50.307849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:50.408587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:50.408547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:55.357656 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:55.357615 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:55.358092 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:55.357718 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:55.362493 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:55.362471 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:55.768210 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:55.768125 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:21:55.817736 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:55.817708 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556d7b4bf4-knb4c"] Jul 28 14:21:55.934333 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:55.934298 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:56.034987 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:56.034887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:21:56.098463 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:21:56.098426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:22:20.837956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:20.837913 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-556d7b4bf4-knb4c" podUID="14eb7585-632d-4e3b-ad3d-2278c3fabcd0" containerName="console" containerID="cri-o://f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a" gracePeriod=15 Jul 28 14:22:21.072397 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.072373 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:22:21.219767 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219653 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-oauth-config\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.219767 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-config\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.219977 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219779 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-serving-cert\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.219977 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219808 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-trusted-ca-bundle\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.219977 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219899 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvhxm\" (UniqueName: \"kubernetes.io/projected/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-kube-api-access-jvhxm\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.219977 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219944 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-service-ca\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.219977 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.219967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-oauth-serving-cert\") pod \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\" (UID: \"14eb7585-632d-4e3b-ad3d-2278c3fabcd0\") " Jul 28 14:22:21.220225 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.220200 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-config" (OuterVolumeSpecName: "console-config") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:22:21.220396 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.220310 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-service-ca" (OuterVolumeSpecName: "service-ca") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:22:21.220541 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.220323 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:22:21.220541 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.220508 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:22:21.221973 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.221948 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:22:21.221973 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.221949 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-kube-api-access-jvhxm" (OuterVolumeSpecName: "kube-api-access-jvhxm") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "kube-api-access-jvhxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:22:21.222129 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.222021 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "14eb7585-632d-4e3b-ad3d-2278c3fabcd0" (UID: "14eb7585-632d-4e3b-ad3d-2278c3fabcd0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:22:21.321164 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321134 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.321164 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321160 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-trusted-ca-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.321164 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321169 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvhxm\" (UniqueName: \"kubernetes.io/projected/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-kube-api-access-jvhxm\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.321399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321178 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.321399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321187 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-oauth-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.321399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321195 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-oauth-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.321399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.321206 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14eb7585-632d-4e3b-ad3d-2278c3fabcd0-console-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:22:21.862373 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.862337 2562 generic.go:356] "Generic (PLEG): container finished" podID="14eb7585-632d-4e3b-ad3d-2278c3fabcd0" containerID="f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a" exitCode=0 Jul 28 14:22:21.862860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.862380 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556d7b4bf4-knb4c" Jul 28 14:22:21.862860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.862414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556d7b4bf4-knb4c" event={"ID":"14eb7585-632d-4e3b-ad3d-2278c3fabcd0","Type":"ContainerDied","Data":"f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a"} Jul 28 14:22:21.862860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.862463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556d7b4bf4-knb4c" event={"ID":"14eb7585-632d-4e3b-ad3d-2278c3fabcd0","Type":"ContainerDied","Data":"4861443f7eb425c22df651e2fd12ab8b0e0e39d2ced84dafebafa207f72de465"} Jul 28 14:22:21.862860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.862492 2562 scope.go:117] "RemoveContainer" containerID="f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a" Jul 28 14:22:21.870867 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.870846 2562 scope.go:117] "RemoveContainer" containerID="f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a" Jul 28 14:22:21.871128 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:22:21.871112 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a\": container with ID starting with f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a not found: ID does not exist" containerID="f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a" Jul 28 14:22:21.871175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.871137 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a"} err="failed to get container status \"f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a\": rpc error: code = NotFound desc = could not find container \"f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a\": container with ID starting with f7713d31fbd1d7a6e587a8410e6186d10fe320f5ae023aeb4ebb9e33b6b0085a not found: ID does not exist" Jul 28 14:22:21.884943 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.884918 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556d7b4bf4-knb4c"] Jul 28 14:22:21.889135 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:21.889114 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556d7b4bf4-knb4c"] Jul 28 14:22:22.887368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:22.887335 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14eb7585-632d-4e3b-ad3d-2278c3fabcd0" path="/var/lib/kubelet/pods/14eb7585-632d-4e3b-ad3d-2278c3fabcd0/volumes" Jul 28 14:22:57.172000 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:57.171956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:22:57.272031 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:57.271996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:22:58.798570 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:58.798489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:22:59.393801 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.393762 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl"] Jul 28 14:22:59.394112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.394093 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14eb7585-632d-4e3b-ad3d-2278c3fabcd0" containerName="console" Jul 28 14:22:59.394190 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.394114 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="14eb7585-632d-4e3b-ad3d-2278c3fabcd0" containerName="console" Jul 28 14:22:59.394355 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.394218 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="14eb7585-632d-4e3b-ad3d-2278c3fabcd0" containerName="console" Jul 28 14:22:59.397357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.397335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.399582 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.399563 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 14:22:59.399693 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.399637 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 14:22:59.400290 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.400274 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kdlt4\"" Jul 28 14:22:59.407663 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.407640 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl"] Jul 28 14:22:59.421053 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.421027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.421182 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.421085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krt2\" (UniqueName: \"kubernetes.io/projected/8dd81628-704f-441f-9dbd-64b9e1722bc6-kube-api-access-6krt2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.421182 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.421122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.521885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.521854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krt2\" (UniqueName: \"kubernetes.io/projected/8dd81628-704f-441f-9dbd-64b9e1722bc6-kube-api-access-6krt2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.522052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.521894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.522052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.521929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.522238 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.522223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.522272 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.522256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.530528 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.530497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krt2\" (UniqueName: \"kubernetes.io/projected/8dd81628-704f-441f-9dbd-64b9e1722bc6-kube-api-access-6krt2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.657007 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.656916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:22:59.706725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.706665 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:22:59.839892 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.839762 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl"] Jul 28 14:22:59.842702 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:22:59.842649 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd81628_704f_441f_9dbd_64b9e1722bc6.slice/crio-bcade8df58184d3edf14361f0bd9757bb3efdd227340774644614868a77bc1ae WatchSource:0}: Error finding container bcade8df58184d3edf14361f0bd9757bb3efdd227340774644614868a77bc1ae: Status 404 returned error can't find the container with id bcade8df58184d3edf14361f0bd9757bb3efdd227340774644614868a77bc1ae Jul 28 14:22:59.993420 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:22:59.993336 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" event={"ID":"8dd81628-704f-441f-9dbd-64b9e1722bc6","Type":"ContainerStarted","Data":"bcade8df58184d3edf14361f0bd9757bb3efdd227340774644614868a77bc1ae"} Jul 28 14:23:00.646013 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:00.645974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:07.021729 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:07.021666 2562 generic.go:356] "Generic (PLEG): container finished" podID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerID="f85496368b30c0f6256b67471e0a915f7504a15c9ee55a9f2608e24b0bfbaa53" exitCode=0 Jul 28 14:23:07.021729 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:07.021714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" event={"ID":"8dd81628-704f-441f-9dbd-64b9e1722bc6","Type":"ContainerDied","Data":"f85496368b30c0f6256b67471e0a915f7504a15c9ee55a9f2608e24b0bfbaa53"} Jul 28 14:23:10.035658 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:10.035623 2562 generic.go:356] "Generic (PLEG): container finished" podID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerID="2c79b4877b9864202c362e250c40c4c89d26f81846b720dbbd88fd6b7c4d1035" exitCode=0 Jul 28 14:23:10.036083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:10.035717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" event={"ID":"8dd81628-704f-441f-9dbd-64b9e1722bc6","Type":"ContainerDied","Data":"2c79b4877b9864202c362e250c40c4c89d26f81846b720dbbd88fd6b7c4d1035"} Jul 28 14:23:17.066733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:17.066687 2562 generic.go:356] "Generic (PLEG): container finished" podID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerID="8c1fd8c6ce286e9538d242f13dbc30982e19e332958887ab970140351ccbdec0" exitCode=0 Jul 28 14:23:17.067116 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:17.066748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" event={"ID":"8dd81628-704f-441f-9dbd-64b9e1722bc6","Type":"ContainerDied","Data":"8c1fd8c6ce286e9538d242f13dbc30982e19e332958887ab970140351ccbdec0"} Jul 28 14:23:18.195614 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.195589 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:23:18.285234 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.285203 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-bundle\") pod \"8dd81628-704f-441f-9dbd-64b9e1722bc6\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " Jul 28 14:23:18.285410 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.285289 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6krt2\" (UniqueName: \"kubernetes.io/projected/8dd81628-704f-441f-9dbd-64b9e1722bc6-kube-api-access-6krt2\") pod \"8dd81628-704f-441f-9dbd-64b9e1722bc6\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " Jul 28 14:23:18.285410 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.285321 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-util\") pod \"8dd81628-704f-441f-9dbd-64b9e1722bc6\" (UID: \"8dd81628-704f-441f-9dbd-64b9e1722bc6\") " Jul 28 14:23:18.285907 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.285881 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-bundle" (OuterVolumeSpecName: "bundle") pod "8dd81628-704f-441f-9dbd-64b9e1722bc6" (UID: "8dd81628-704f-441f-9dbd-64b9e1722bc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:23:18.287459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.287426 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd81628-704f-441f-9dbd-64b9e1722bc6-kube-api-access-6krt2" (OuterVolumeSpecName: "kube-api-access-6krt2") pod "8dd81628-704f-441f-9dbd-64b9e1722bc6" (UID: "8dd81628-704f-441f-9dbd-64b9e1722bc6"). InnerVolumeSpecName "kube-api-access-6krt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:23:18.289667 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.289646 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-util" (OuterVolumeSpecName: "util") pod "8dd81628-704f-441f-9dbd-64b9e1722bc6" (UID: "8dd81628-704f-441f-9dbd-64b9e1722bc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:23:18.386212 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.386139 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krt2\" (UniqueName: \"kubernetes.io/projected/8dd81628-704f-441f-9dbd-64b9e1722bc6-kube-api-access-6krt2\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:18.386212 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.386166 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:18.386212 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.386175 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8dd81628-704f-441f-9dbd-64b9e1722bc6-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:18.762291 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.762202 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:23:18.766589 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.766567 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:23:18.767334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:18.767307 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 14:23:19.075573 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:19.075550 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" Jul 28 14:23:19.075786 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:19.075545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407zbsl" event={"ID":"8dd81628-704f-441f-9dbd-64b9e1722bc6","Type":"ContainerDied","Data":"bcade8df58184d3edf14361f0bd9757bb3efdd227340774644614868a77bc1ae"} Jul 28 14:23:19.075786 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:19.075670 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcade8df58184d3edf14361f0bd9757bb3efdd227340774644614868a77bc1ae" Jul 28 14:23:20.450583 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:20.450545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:21.271366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:21.271326 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:22.638608 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638571 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92"] Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638855 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="util" Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638866 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="util" Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638874 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="pull" Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638879 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="pull" Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638887 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="extract" Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638893 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="extract" Jul 28 14:23:22.638999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.638942 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd81628-704f-441f-9dbd-64b9e1722bc6" containerName="extract" Jul 28 14:23:22.689200 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.689169 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92"] Jul 28 14:23:22.689349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.689285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.692835 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.692810 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hb7b7\"" Jul 28 14:23:22.693352 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.693334 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 14:23:22.693691 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.693659 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 14:23:22.819029 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.818998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78sn\" (UniqueName: \"kubernetes.io/projected/16851ce1-eeff-4f48-b957-a871c7863e34-kube-api-access-h78sn\") pod \"cert-manager-operator-controller-manager-55c444bf86-ltw92\" (UID: \"16851ce1-eeff-4f48-b957-a871c7863e34\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.819175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.819099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16851ce1-eeff-4f48-b957-a871c7863e34-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ltw92\" (UID: \"16851ce1-eeff-4f48-b957-a871c7863e34\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.920289 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.920211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16851ce1-eeff-4f48-b957-a871c7863e34-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ltw92\" (UID: \"16851ce1-eeff-4f48-b957-a871c7863e34\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.920289 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.920252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78sn\" (UniqueName: \"kubernetes.io/projected/16851ce1-eeff-4f48-b957-a871c7863e34-kube-api-access-h78sn\") pod \"cert-manager-operator-controller-manager-55c444bf86-ltw92\" (UID: \"16851ce1-eeff-4f48-b957-a871c7863e34\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.920587 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.920567 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16851ce1-eeff-4f48-b957-a871c7863e34-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ltw92\" (UID: \"16851ce1-eeff-4f48-b957-a871c7863e34\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.930273 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.930253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78sn\" (UniqueName: \"kubernetes.io/projected/16851ce1-eeff-4f48-b957-a871c7863e34-kube-api-access-h78sn\") pod \"cert-manager-operator-controller-manager-55c444bf86-ltw92\" (UID: \"16851ce1-eeff-4f48-b957-a871c7863e34\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:22.998534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:22.998496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" Jul 28 14:23:23.154545 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:23.154521 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92"] Jul 28 14:23:23.157189 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:23:23.157142 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16851ce1_eeff_4f48_b957_a871c7863e34.slice/crio-b51cd235a70f6844ab68a39d9558e3387bca54abf7468670c459b4b6f8ef33d9 WatchSource:0}: Error finding container b51cd235a70f6844ab68a39d9558e3387bca54abf7468670c459b4b6f8ef33d9: Status 404 returned error can't find the container with id b51cd235a70f6844ab68a39d9558e3387bca54abf7468670c459b4b6f8ef33d9 Jul 28 14:23:23.160135 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:23.160116 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 14:23:24.095530 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:24.095495 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" event={"ID":"16851ce1-eeff-4f48-b957-a871c7863e34","Type":"ContainerStarted","Data":"b51cd235a70f6844ab68a39d9558e3387bca54abf7468670c459b4b6f8ef33d9"} Jul 28 14:23:24.256382 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:24.256343 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:24.314623 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:24.314579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:26.064548 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:26.064516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:26.131962 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:26.131929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:26.988754 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:26.988717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:27.089637 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.089599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:27.108866 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.108837 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tmhn2"] Jul 28 14:23:27.133140 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.133113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" event={"ID":"16851ce1-eeff-4f48-b957-a871c7863e34","Type":"ContainerStarted","Data":"50fe8c8d9774eeb0a2af9e8eafd35f93a7e673ec0b8a7883021cde30ef61b65d"} Jul 28 14:23:27.133275 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.133152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tmhn2"] Jul 28 14:23:27.133275 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.133226 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.135579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.135559 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 14:23:27.135718 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.135565 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 14:23:27.143450 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.143383 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ltw92" podStartSLOduration=2.075764208 podStartE2EDuration="5.143371575s" totalImagesPullingTime="3.067607367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:27.141913776 +0000 UTC m=+308.807155598" watchObservedRunningTime="2026-07-28 14:23:27.143371575 +0000 UTC m=+308.808613399" Jul 28 14:23:27.255351 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.255301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hmml\" (UniqueName: \"kubernetes.io/projected/e0b6efd0-c123-41de-af9b-63808bdbd175-kube-api-access-9hmml\") pod \"cert-manager-webhook-6486ffd997-tmhn2\" (UID: \"e0b6efd0-c123-41de-af9b-63808bdbd175\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.255351 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.255341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0b6efd0-c123-41de-af9b-63808bdbd175-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tmhn2\" (UID: \"e0b6efd0-c123-41de-af9b-63808bdbd175\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.356584 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.356566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hmml\" (UniqueName: \"kubernetes.io/projected/e0b6efd0-c123-41de-af9b-63808bdbd175-kube-api-access-9hmml\") pod \"cert-manager-webhook-6486ffd997-tmhn2\" (UID: \"e0b6efd0-c123-41de-af9b-63808bdbd175\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.356702 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.356602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0b6efd0-c123-41de-af9b-63808bdbd175-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tmhn2\" (UID: \"e0b6efd0-c123-41de-af9b-63808bdbd175\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.369912 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.369893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0b6efd0-c123-41de-af9b-63808bdbd175-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tmhn2\" (UID: \"e0b6efd0-c123-41de-af9b-63808bdbd175\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.370007 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.369977 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hmml\" (UniqueName: \"kubernetes.io/projected/e0b6efd0-c123-41de-af9b-63808bdbd175-kube-api-access-9hmml\") pod \"cert-manager-webhook-6486ffd997-tmhn2\" (UID: \"e0b6efd0-c123-41de-af9b-63808bdbd175\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.453454 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.453439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:27.627519 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.627477 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tmhn2"] Jul 28 14:23:27.629705 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:23:27.629666 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0b6efd0_c123_41de_af9b_63808bdbd175.slice/crio-edd32fb8f1b062385563bbced0cab047df94bbbe3e04c21db7da1388297de623 WatchSource:0}: Error finding container edd32fb8f1b062385563bbced0cab047df94bbbe3e04c21db7da1388297de623: Status 404 returned error can't find the container with id edd32fb8f1b062385563bbced0cab047df94bbbe3e04c21db7da1388297de623 Jul 28 14:23:27.674516 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.674489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:27.830471 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.830442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:27.854282 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.854251 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l"] Jul 28 14:23:27.857374 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.857355 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:27.860760 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.860743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kdlt4\"" Jul 28 14:23:27.861025 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.861007 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 14:23:27.861025 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.861015 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 14:23:27.898877 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.898857 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l"] Jul 28 14:23:27.960044 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.960018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:27.960127 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.960046 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:27.960127 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:27.960075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5schg\" (UniqueName: \"kubernetes.io/projected/dea206b6-447c-4cec-a63b-df8aad68f93a-kube-api-access-5schg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.060597 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.060573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.060743 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.060601 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.060743 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.060649 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5schg\" (UniqueName: \"kubernetes.io/projected/dea206b6-447c-4cec-a63b-df8aad68f93a-kube-api-access-5schg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.061029 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.061006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.061090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.061024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.083311 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.083261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5schg\" (UniqueName: \"kubernetes.io/projected/dea206b6-447c-4cec-a63b-df8aad68f93a-kube-api-access-5schg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.119885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.119859 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" event={"ID":"e0b6efd0-c123-41de-af9b-63808bdbd175","Type":"ContainerStarted","Data":"edd32fb8f1b062385563bbced0cab047df94bbbe3e04c21db7da1388297de623"} Jul 28 14:23:28.166337 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.166317 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:28.310790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:28.310572 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l"] Jul 28 14:23:28.313347 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:23:28.313320 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea206b6_447c_4cec_a63b_df8aad68f93a.slice/crio-d96c9f0e8d13542920a4c12469393010c08791670f7308c2dffdc8710bd7778c WatchSource:0}: Error finding container d96c9f0e8d13542920a4c12469393010c08791670f7308c2dffdc8710bd7778c: Status 404 returned error can't find the container with id d96c9f0e8d13542920a4c12469393010c08791670f7308c2dffdc8710bd7778c Jul 28 14:23:29.127472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:29.127438 2562 generic.go:356] "Generic (PLEG): container finished" podID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerID="be657a9776472e58dbfe36c802b4d5cf3f33de516add7aa50e0ab5650ae05f27" exitCode=0 Jul 28 14:23:29.127930 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:29.127486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" event={"ID":"dea206b6-447c-4cec-a63b-df8aad68f93a","Type":"ContainerDied","Data":"be657a9776472e58dbfe36c802b4d5cf3f33de516add7aa50e0ab5650ae05f27"} Jul 28 14:23:29.127930 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:29.127531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" event={"ID":"dea206b6-447c-4cec-a63b-df8aad68f93a","Type":"ContainerStarted","Data":"d96c9f0e8d13542920a4c12469393010c08791670f7308c2dffdc8710bd7778c"} Jul 28 14:23:30.623604 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:30.623459 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:30.723230 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:30.723147 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:31.138502 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:31.138466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" event={"ID":"e0b6efd0-c123-41de-af9b-63808bdbd175","Type":"ContainerStarted","Data":"431f4d2c82cc165fa979e72f0d3db9209faf6d495fa4f78a6777e28d9a73fada"} Jul 28 14:23:31.138720 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:31.138604 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:31.162241 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:31.162192 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" podStartSLOduration=1.338141332 podStartE2EDuration="4.162177219s" totalImagesPullingTime="2.824035887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:31.159776416 +0000 UTC m=+312.825018238" watchObservedRunningTime="2026-07-28 14:23:31.162177219 +0000 UTC m=+312.827419042" Jul 28 14:23:32.144575 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:32.144479 2562 generic.go:356] "Generic (PLEG): container finished" podID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerID="c4adb74f0dd4e75027e3d30a7ec2e76e41d55dc38efd4e95ad427161e8cfae0b" exitCode=0 Jul 28 14:23:32.145053 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:32.144570 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" event={"ID":"dea206b6-447c-4cec-a63b-df8aad68f93a","Type":"ContainerDied","Data":"c4adb74f0dd4e75027e3d30a7ec2e76e41d55dc38efd4e95ad427161e8cfae0b"} Jul 28 14:23:33.151324 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:33.151287 2562 generic.go:356] "Generic (PLEG): container finished" podID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerID="e6f5410ca0ade08137e02cc16b9a692063ecf9fda41b5d2f8827bf508311e8f1" exitCode=0 Jul 28 14:23:33.151740 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:33.151370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" event={"ID":"dea206b6-447c-4cec-a63b-df8aad68f93a","Type":"ContainerDied","Data":"e6f5410ca0ade08137e02cc16b9a692063ecf9fda41b5d2f8827bf508311e8f1"} Jul 28 14:23:34.280415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.280390 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:34.411490 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.411400 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5schg\" (UniqueName: \"kubernetes.io/projected/dea206b6-447c-4cec-a63b-df8aad68f93a-kube-api-access-5schg\") pod \"dea206b6-447c-4cec-a63b-df8aad68f93a\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " Jul 28 14:23:34.411490 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.411465 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-bundle\") pod \"dea206b6-447c-4cec-a63b-df8aad68f93a\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " Jul 28 14:23:34.411814 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.411542 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-util\") pod \"dea206b6-447c-4cec-a63b-df8aad68f93a\" (UID: \"dea206b6-447c-4cec-a63b-df8aad68f93a\") " Jul 28 14:23:34.411978 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.411944 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-bundle" (OuterVolumeSpecName: "bundle") pod "dea206b6-447c-4cec-a63b-df8aad68f93a" (UID: "dea206b6-447c-4cec-a63b-df8aad68f93a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:23:34.413502 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.413481 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea206b6-447c-4cec-a63b-df8aad68f93a-kube-api-access-5schg" (OuterVolumeSpecName: "kube-api-access-5schg") pod "dea206b6-447c-4cec-a63b-df8aad68f93a" (UID: "dea206b6-447c-4cec-a63b-df8aad68f93a"). InnerVolumeSpecName "kube-api-access-5schg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:23:34.416389 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.416362 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-util" (OuterVolumeSpecName: "util") pod "dea206b6-447c-4cec-a63b-df8aad68f93a" (UID: "dea206b6-447c-4cec-a63b-df8aad68f93a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:23:34.513094 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.513055 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:34.513094 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.513088 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5schg\" (UniqueName: \"kubernetes.io/projected/dea206b6-447c-4cec-a63b-df8aad68f93a-kube-api-access-5schg\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:34.513094 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.513102 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dea206b6-447c-4cec-a63b-df8aad68f93a-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:34.618773 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.618737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:34.697900 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:34.697804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:35.163767 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:35.163733 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" event={"ID":"dea206b6-447c-4cec-a63b-df8aad68f93a","Type":"ContainerDied","Data":"d96c9f0e8d13542920a4c12469393010c08791670f7308c2dffdc8710bd7778c"} Jul 28 14:23:35.163932 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:35.163783 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d96c9f0e8d13542920a4c12469393010c08791670f7308c2dffdc8710bd7778c" Jul 28 14:23:35.163932 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:35.163745 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd7w5l" Jul 28 14:23:37.147318 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:37.147287 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tmhn2" Jul 28 14:23:37.810153 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:37.810110 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:38.348913 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.348874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:38.446701 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.446651 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:38.923821 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.923787 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9bfdr"] Jul 28 14:23:38.924061 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924050 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="pull" Jul 28 14:23:38.924117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924063 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="pull" Jul 28 14:23:38.924117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924073 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="extract" Jul 28 14:23:38.924117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924079 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="extract" Jul 28 14:23:38.924117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924097 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="util" Jul 28 14:23:38.924117 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924102 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="util" Jul 28 14:23:38.924275 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.924148 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea206b6-447c-4cec-a63b-df8aad68f93a" containerName="extract" Jul 28 14:23:38.927993 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.927975 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:38.930837 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.930820 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-49grn\"" Jul 28 14:23:38.939241 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:38.939217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9bfdr"] Jul 28 14:23:39.049966 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.049933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0264a7b6-9590-4fca-8c4f-12804906dac7-bound-sa-token\") pod \"cert-manager-55b4d954d5-9bfdr\" (UID: \"0264a7b6-9590-4fca-8c4f-12804906dac7\") " pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.050143 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.049973 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkfzf\" (UniqueName: \"kubernetes.io/projected/0264a7b6-9590-4fca-8c4f-12804906dac7-kube-api-access-jkfzf\") pod \"cert-manager-55b4d954d5-9bfdr\" (UID: \"0264a7b6-9590-4fca-8c4f-12804906dac7\") " pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.150391 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.150345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0264a7b6-9590-4fca-8c4f-12804906dac7-bound-sa-token\") pod \"cert-manager-55b4d954d5-9bfdr\" (UID: \"0264a7b6-9590-4fca-8c4f-12804906dac7\") " pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.150564 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.150409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkfzf\" (UniqueName: \"kubernetes.io/projected/0264a7b6-9590-4fca-8c4f-12804906dac7-kube-api-access-jkfzf\") pod \"cert-manager-55b4d954d5-9bfdr\" (UID: \"0264a7b6-9590-4fca-8c4f-12804906dac7\") " pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.160580 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.160540 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0264a7b6-9590-4fca-8c4f-12804906dac7-bound-sa-token\") pod \"cert-manager-55b4d954d5-9bfdr\" (UID: \"0264a7b6-9590-4fca-8c4f-12804906dac7\") " pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.160774 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.160641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkfzf\" (UniqueName: \"kubernetes.io/projected/0264a7b6-9590-4fca-8c4f-12804906dac7-kube-api-access-jkfzf\") pod \"cert-manager-55b4d954d5-9bfdr\" (UID: \"0264a7b6-9590-4fca-8c4f-12804906dac7\") " pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.236224 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.236154 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-9bfdr" Jul 28 14:23:39.365172 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:39.365147 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-9bfdr"] Jul 28 14:23:39.367654 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:23:39.367618 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0264a7b6_9590_4fca_8c4f_12804906dac7.slice/crio-6be3a3b4e9ccc1ad3ed8be48eeaf935e24c78c931450214dbf936650c95d57eb WatchSource:0}: Error finding container 6be3a3b4e9ccc1ad3ed8be48eeaf935e24c78c931450214dbf936650c95d57eb: Status 404 returned error can't find the container with id 6be3a3b4e9ccc1ad3ed8be48eeaf935e24c78c931450214dbf936650c95d57eb Jul 28 14:23:40.176835 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.176801 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828"] Jul 28 14:23:40.180403 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.180383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.182920 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.182898 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 14:23:40.183041 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.182899 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 14:23:40.183659 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.183639 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-w2bbc\"" Jul 28 14:23:40.187201 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.187176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9bfdr" event={"ID":"0264a7b6-9590-4fca-8c4f-12804906dac7","Type":"ContainerStarted","Data":"80d9e9182b9c9b6e8375c675cf69dec2eecffe99f2aba05625becee37250ca4a"} Jul 28 14:23:40.187306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.187213 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-9bfdr" event={"ID":"0264a7b6-9590-4fca-8c4f-12804906dac7","Type":"ContainerStarted","Data":"6be3a3b4e9ccc1ad3ed8be48eeaf935e24c78c931450214dbf936650c95d57eb"} Jul 28 14:23:40.190430 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.190408 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828"] Jul 28 14:23:40.221241 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.221169 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-9bfdr" podStartSLOduration=2.221145125 podStartE2EDuration="2.221145125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:40.220926668 +0000 UTC m=+321.886168492" watchObservedRunningTime="2026-07-28 14:23:40.221145125 +0000 UTC m=+321.886386948" Jul 28 14:23:40.361464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.361431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580b1b4d-db98-4b1f-9d3b-66f4d485c9c2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hx828\" (UID: \"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.361464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.361464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lz5\" (UniqueName: \"kubernetes.io/projected/580b1b4d-db98-4b1f-9d3b-66f4d485c9c2-kube-api-access-j9lz5\") pod \"openshift-lws-operator-fd8ccff4c-hx828\" (UID: \"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.462205 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.462144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580b1b4d-db98-4b1f-9d3b-66f4d485c9c2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hx828\" (UID: \"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.462205 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.462173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lz5\" (UniqueName: \"kubernetes.io/projected/580b1b4d-db98-4b1f-9d3b-66f4d485c9c2-kube-api-access-j9lz5\") pod \"openshift-lws-operator-fd8ccff4c-hx828\" (UID: \"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.462534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.462485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/580b1b4d-db98-4b1f-9d3b-66f4d485c9c2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hx828\" (UID: \"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.483990 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.483959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lz5\" (UniqueName: \"kubernetes.io/projected/580b1b4d-db98-4b1f-9d3b-66f4d485c9c2-kube-api-access-j9lz5\") pod \"openshift-lws-operator-fd8ccff4c-hx828\" (UID: \"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.490690 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.490661 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" Jul 28 14:23:40.615529 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:40.615505 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828"] Jul 28 14:23:40.617816 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:23:40.617792 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod580b1b4d_db98_4b1f_9d3b_66f4d485c9c2.slice/crio-f5599fe3cea3705d9ef485c0ffa31e5bac76e3d6f697e705a6b5fd6895a83abb WatchSource:0}: Error finding container f5599fe3cea3705d9ef485c0ffa31e5bac76e3d6f697e705a6b5fd6895a83abb: Status 404 returned error can't find the container with id f5599fe3cea3705d9ef485c0ffa31e5bac76e3d6f697e705a6b5fd6895a83abb Jul 28 14:23:41.192944 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:41.192907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" event={"ID":"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2","Type":"ContainerStarted","Data":"f5599fe3cea3705d9ef485c0ffa31e5bac76e3d6f697e705a6b5fd6895a83abb"} Jul 28 14:23:43.203892 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:43.203861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" event={"ID":"580b1b4d-db98-4b1f-9d3b-66f4d485c9c2","Type":"ContainerStarted","Data":"925b26a65730bad9461b5294296da4eae05e66157dbee0c4296ef91dfb5eb2d0"} Jul 28 14:23:43.222720 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:43.222648 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hx828" podStartSLOduration=1.435432926 podStartE2EDuration="3.222633304s" totalImagesPullingTime="1.787200378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:23:43.220971127 +0000 UTC m=+324.886212952" watchObservedRunningTime="2026-07-28 14:23:43.222633304 +0000 UTC m=+324.887875130" Jul 28 14:23:43.955770 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:43.955730 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:44.055506 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:44.055464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:46.004720 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.004655 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4"] Jul 28 14:23:46.008359 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.008338 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.011174 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.011144 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kdlt4\"" Jul 28 14:23:46.011308 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.011217 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 14:23:46.012431 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.012408 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 14:23:46.019578 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.019545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4"] Jul 28 14:23:46.105640 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.105610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.105782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.105646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.105782 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.105696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j8kq\" (UniqueName: \"kubernetes.io/projected/0d73d271-0dfb-4faf-b36b-f575d523e10d-kube-api-access-9j8kq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.196989 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.196958 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:23:46.206009 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.205981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.206141 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.206015 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.206141 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.206040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j8kq\" (UniqueName: \"kubernetes.io/projected/0d73d271-0dfb-4faf-b36b-f575d523e10d-kube-api-access-9j8kq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.206353 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.206333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.206428 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.206401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.217361 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.217336 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j8kq\" (UniqueName: \"kubernetes.io/projected/0d73d271-0dfb-4faf-b36b-f575d523e10d-kube-api-access-9j8kq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.324710 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.324668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:46.451481 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:46.451454 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4"] Jul 28 14:23:46.453164 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:23:46.453128 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d73d271_0dfb_4faf_b36b_f575d523e10d.slice/crio-560e5fdcd38dcac6d8964a57ec02a284534535d783a6334fe845b5f7c4cc8827 WatchSource:0}: Error finding container 560e5fdcd38dcac6d8964a57ec02a284534535d783a6334fe845b5f7c4cc8827: Status 404 returned error can't find the container with id 560e5fdcd38dcac6d8964a57ec02a284534535d783a6334fe845b5f7c4cc8827 Jul 28 14:23:47.223664 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:47.223627 2562 generic.go:356] "Generic (PLEG): container finished" podID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerID="882ad5ea03ed9f1e56553f48ee50f80931001d3b2d1591f50263652a3404fc5a" exitCode=0 Jul 28 14:23:47.224013 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:47.223729 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" event={"ID":"0d73d271-0dfb-4faf-b36b-f575d523e10d","Type":"ContainerDied","Data":"882ad5ea03ed9f1e56553f48ee50f80931001d3b2d1591f50263652a3404fc5a"} Jul 28 14:23:47.224013 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:47.223795 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" event={"ID":"0d73d271-0dfb-4faf-b36b-f575d523e10d","Type":"ContainerStarted","Data":"560e5fdcd38dcac6d8964a57ec02a284534535d783a6334fe845b5f7c4cc8827"} Jul 28 14:23:49.233979 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:49.233944 2562 generic.go:356] "Generic (PLEG): container finished" podID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerID="d0bff6894b6f99d7293420968061a2ccae799465a6bde96483d158340a93db18" exitCode=0 Jul 28 14:23:49.234465 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:49.234049 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" event={"ID":"0d73d271-0dfb-4faf-b36b-f575d523e10d","Type":"ContainerDied","Data":"d0bff6894b6f99d7293420968061a2ccae799465a6bde96483d158340a93db18"} Jul 28 14:23:50.240499 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:50.240463 2562 generic.go:356] "Generic (PLEG): container finished" podID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerID="99d0edfc5f1a06fc73a0cf93ee6cea5e61959cf57ced5a2aed31e5c3a6d39d16" exitCode=0 Jul 28 14:23:50.240889 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:50.240504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" event={"ID":"0d73d271-0dfb-4faf-b36b-f575d523e10d","Type":"ContainerDied","Data":"99d0edfc5f1a06fc73a0cf93ee6cea5e61959cf57ced5a2aed31e5c3a6d39d16"} Jul 28 14:23:51.375592 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.375568 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:51.443265 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.443245 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-bundle\") pod \"0d73d271-0dfb-4faf-b36b-f575d523e10d\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " Jul 28 14:23:51.443388 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.443276 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j8kq\" (UniqueName: \"kubernetes.io/projected/0d73d271-0dfb-4faf-b36b-f575d523e10d-kube-api-access-9j8kq\") pod \"0d73d271-0dfb-4faf-b36b-f575d523e10d\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " Jul 28 14:23:51.443388 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.443343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-util\") pod \"0d73d271-0dfb-4faf-b36b-f575d523e10d\" (UID: \"0d73d271-0dfb-4faf-b36b-f575d523e10d\") " Jul 28 14:23:51.444040 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.444016 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-bundle" (OuterVolumeSpecName: "bundle") pod "0d73d271-0dfb-4faf-b36b-f575d523e10d" (UID: "0d73d271-0dfb-4faf-b36b-f575d523e10d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:23:51.445298 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.445272 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d73d271-0dfb-4faf-b36b-f575d523e10d-kube-api-access-9j8kq" (OuterVolumeSpecName: "kube-api-access-9j8kq") pod "0d73d271-0dfb-4faf-b36b-f575d523e10d" (UID: "0d73d271-0dfb-4faf-b36b-f575d523e10d"). InnerVolumeSpecName "kube-api-access-9j8kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:23:51.448484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.448464 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-util" (OuterVolumeSpecName: "util") pod "0d73d271-0dfb-4faf-b36b-f575d523e10d" (UID: "0d73d271-0dfb-4faf-b36b-f575d523e10d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:23:51.544290 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.544271 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:51.544373 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.544291 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d73d271-0dfb-4faf-b36b-f575d523e10d-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:51.544373 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:51.544301 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j8kq\" (UniqueName: \"kubernetes.io/projected/0d73d271-0dfb-4faf-b36b-f575d523e10d-kube-api-access-9j8kq\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:23:52.251668 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:52.251641 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" Jul 28 14:23:52.251834 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:52.251646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcx5s4" event={"ID":"0d73d271-0dfb-4faf-b36b-f575d523e10d","Type":"ContainerDied","Data":"560e5fdcd38dcac6d8964a57ec02a284534535d783a6334fe845b5f7c4cc8827"} Jul 28 14:23:52.251834 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:23:52.251721 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="560e5fdcd38dcac6d8964a57ec02a284534535d783a6334fe845b5f7c4cc8827" Jul 28 14:24:02.495264 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.495230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:02.622259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622228 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9"] Jul 28 14:24:02.622708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622689 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="extract" Jul 28 14:24:02.622708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622710 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="extract" Jul 28 14:24:02.622790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622729 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="pull" Jul 28 14:24:02.622790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622738 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="pull" Jul 28 14:24:02.622790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622754 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="util" Jul 28 14:24:02.622790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622763 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="util" Jul 28 14:24:02.622910 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.622834 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d73d271-0dfb-4faf-b36b-f575d523e10d" containerName="extract" Jul 28 14:24:02.627319 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.627299 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.630772 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.630749 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 14:24:02.631006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.630988 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4xvsd\"" Jul 28 14:24:02.631641 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.631620 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 14:24:02.631641 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.631636 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 14:24:02.641510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.641479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9"] Jul 28 14:24:02.721321 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.721292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/773f393a-bdd0-4246-90f8-1610375f05be-manager-config\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.721438 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.721325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/773f393a-bdd0-4246-90f8-1610375f05be-metrics-cert\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.721438 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.721346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzn9q\" (UniqueName: \"kubernetes.io/projected/773f393a-bdd0-4246-90f8-1610375f05be-kube-api-access-tzn9q\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.721438 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.721403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/773f393a-bdd0-4246-90f8-1610375f05be-cert\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.822737 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.822707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/773f393a-bdd0-4246-90f8-1610375f05be-manager-config\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.822860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.822744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/773f393a-bdd0-4246-90f8-1610375f05be-metrics-cert\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.822860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.822760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzn9q\" (UniqueName: \"kubernetes.io/projected/773f393a-bdd0-4246-90f8-1610375f05be-kube-api-access-tzn9q\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.822860 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.822792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/773f393a-bdd0-4246-90f8-1610375f05be-cert\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.823350 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.823315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/773f393a-bdd0-4246-90f8-1610375f05be-manager-config\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.825121 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.825100 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/773f393a-bdd0-4246-90f8-1610375f05be-cert\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.825221 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.825202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/773f393a-bdd0-4246-90f8-1610375f05be-metrics-cert\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.831566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.831545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzn9q\" (UniqueName: \"kubernetes.io/projected/773f393a-bdd0-4246-90f8-1610375f05be-kube-api-access-tzn9q\") pod \"lws-controller-manager-5d8c94b858-p9br9\" (UID: \"773f393a-bdd0-4246-90f8-1610375f05be\") " pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:02.936502 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:02.936480 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:03.141652 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.141623 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9"] Jul 28 14:24:03.141807 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:24:03.141790 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773f393a_bdd0_4246_90f8_1610375f05be.slice/crio-7983effff959b5a64c929f5906d00588830c8aa6b60f0310daad973f89bcc58c WatchSource:0}: Error finding container 7983effff959b5a64c929f5906d00588830c8aa6b60f0310daad973f89bcc58c: Status 404 returned error can't find the container with id 7983effff959b5a64c929f5906d00588830c8aa6b60f0310daad973f89bcc58c Jul 28 14:24:03.304877 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.304848 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" event={"ID":"773f393a-bdd0-4246-90f8-1610375f05be","Type":"ContainerStarted","Data":"7983effff959b5a64c929f5906d00588830c8aa6b60f0310daad973f89bcc58c"} Jul 28 14:24:03.846526 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.846488 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5"] Jul 28 14:24:03.851070 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.851052 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:03.853410 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.853384 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-htct2\"" Jul 28 14:24:03.853731 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.853710 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 14:24:03.853830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.853748 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 14:24:03.853830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.853752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 14:24:03.854065 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.854049 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 14:24:03.879411 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.879378 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5"] Jul 28 14:24:03.931604 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.931579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52fff165-aeb1-4ef7-b579-1016e42c6293-manifests\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:03.931719 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.931614 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52fff165-aeb1-4ef7-b579-1016e42c6293-tmp\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:03.931719 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.931695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwhh\" (UniqueName: \"kubernetes.io/projected/52fff165-aeb1-4ef7-b579-1016e42c6293-kube-api-access-szwhh\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:03.931795 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.931728 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52fff165-aeb1-4ef7-b579-1016e42c6293-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:03.931795 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:03.931750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52fff165-aeb1-4ef7-b579-1016e42c6293-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.032230 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.032204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwhh\" (UniqueName: \"kubernetes.io/projected/52fff165-aeb1-4ef7-b579-1016e42c6293-kube-api-access-szwhh\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.032348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.032237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52fff165-aeb1-4ef7-b579-1016e42c6293-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.032348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.032260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52fff165-aeb1-4ef7-b579-1016e42c6293-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.032461 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.032354 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52fff165-aeb1-4ef7-b579-1016e42c6293-manifests\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.032461 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.032380 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52fff165-aeb1-4ef7-b579-1016e42c6293-tmp\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.032822 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.032804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52fff165-aeb1-4ef7-b579-1016e42c6293-manifests\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.034687 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.034654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52fff165-aeb1-4ef7-b579-1016e42c6293-tmp\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.034911 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.034894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52fff165-aeb1-4ef7-b579-1016e42c6293-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.034956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.034899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52fff165-aeb1-4ef7-b579-1016e42c6293-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.040591 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.040568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwhh\" (UniqueName: \"kubernetes.io/projected/52fff165-aeb1-4ef7-b579-1016e42c6293-kube-api-access-szwhh\") pod \"opendatahub-operator-controller-manager-65f7956c87-wdnp5\" (UID: \"52fff165-aeb1-4ef7-b579-1016e42c6293\") " pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.161642 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.161592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:04.328930 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:04.328900 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5"] Jul 28 14:24:04.329200 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:24:04.329176 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fff165_aeb1_4ef7_b579_1016e42c6293.slice/crio-89dc057ac8f4f3a4de760323945b7eecca3f01a2507539977c1a3b91a02370e0 WatchSource:0}: Error finding container 89dc057ac8f4f3a4de760323945b7eecca3f01a2507539977c1a3b91a02370e0: Status 404 returned error can't find the container with id 89dc057ac8f4f3a4de760323945b7eecca3f01a2507539977c1a3b91a02370e0 Jul 28 14:24:05.326904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:05.326861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" event={"ID":"52fff165-aeb1-4ef7-b579-1016e42c6293","Type":"ContainerStarted","Data":"89dc057ac8f4f3a4de760323945b7eecca3f01a2507539977c1a3b91a02370e0"} Jul 28 14:24:05.343326 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:05.343296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:06.334209 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:06.334170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" event={"ID":"773f393a-bdd0-4246-90f8-1610375f05be","Type":"ContainerStarted","Data":"97ff3648d5b07ea9a37cc0e6a79ed729a99a3a37f1ace28d9c65a9425c228ba8"} Jul 28 14:24:06.334653 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:06.334261 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:06.352514 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:06.352447 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" podStartSLOduration=2.006764589 podStartE2EDuration="4.352428018s" totalImagesPullingTime="2.345663429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:06.352006929 +0000 UTC m=+348.017248752" watchObservedRunningTime="2026-07-28 14:24:06.352428018 +0000 UTC m=+348.017669842" Jul 28 14:24:08.344341 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:08.344298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" event={"ID":"52fff165-aeb1-4ef7-b579-1016e42c6293","Type":"ContainerStarted","Data":"bae788cd7f5a8e2efb1505e2e101f7d7c8f6bcde410e640b920cbb554a299566"} Jul 28 14:24:08.788976 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:08.788948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:08.867688 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:08.867648 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:09.350176 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:09.350148 2562 generic.go:356] "Generic (PLEG): container finished" podID="52fff165-aeb1-4ef7-b579-1016e42c6293" containerID="bae788cd7f5a8e2efb1505e2e101f7d7c8f6bcde410e640b920cbb554a299566" exitCode=0 Jul 28 14:24:09.350567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:09.350222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" event={"ID":"52fff165-aeb1-4ef7-b579-1016e42c6293","Type":"ContainerDied","Data":"bae788cd7f5a8e2efb1505e2e101f7d7c8f6bcde410e640b920cbb554a299566"} Jul 28 14:24:10.358172 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:10.358121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" event={"ID":"52fff165-aeb1-4ef7-b579-1016e42c6293","Type":"ContainerStarted","Data":"16adc93af0665908766987c799a6fda5379416e8d14c7f0a2bb5725aea0efcbd"} Jul 28 14:24:10.358579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:10.358324 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:10.383579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:10.383524 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" podStartSLOduration=2.60835589 podStartE2EDuration="7.383508142s" totalImagesPullingTime="3.70237956s" totalInitContainerRuntime="1.072772692s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:10.382370939 +0000 UTC m=+352.047612761" watchObservedRunningTime="2026-07-28 14:24:10.383508142 +0000 UTC m=+352.048749965" Jul 28 14:24:15.521716 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:15.521662 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:16.254332 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:16.254298 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:17.340842 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:17.340804 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d8c94b858-p9br9" Jul 28 14:24:21.365272 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:21.365220 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65f7956c87-wdnp5" Jul 28 14:24:23.619191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:23.619154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:23.847565 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:23.847530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:24.249626 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:24.249595 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:24.413809 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:24.413776 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:24.596771 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:24.596742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:25.194511 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:25.194483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:25.488047 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:25.487978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:25.601163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:25.601130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:25.827488 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:25.827458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:25.923170 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:25.923145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:26.210742 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:26.210652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:27.226477 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:27.226442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:31.679125 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:31.679084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:31.788537 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:31.788493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:31.986763 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:31.986689 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l"] Jul 28 14:24:31.995589 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:31.995571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:31.998551 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:31.998532 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nxnmm\"" Jul 28 14:24:31.998899 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:31.998878 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 14:24:32.003304 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.003283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l"] Jul 28 14:24:32.022754 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.022730 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:32.045357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045446 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045446 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045446 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045427 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f669c2-c88e-48b1-9b6d-c84b44878a95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045598 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045480 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045598 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045598 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045722 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkv8\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-kube-api-access-lfkv8\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.045722 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.045646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146039 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkv8\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-kube-api-access-lfkv8\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146371 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146371 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146371 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146371 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f669c2-c88e-48b1-9b6d-c84b44878a95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146580 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146441 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146580 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146769 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146745 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.146978 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.146959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f669c2-c88e-48b1-9b6d-c84b44878a95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.148603 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.148584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.148690 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.148634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.155273 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.155248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkv8\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-kube-api-access-lfkv8\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.155384 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.155259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.306059 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.306037 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:32.483023 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:32.482994 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l"] Jul 28 14:24:32.490259 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:24:32.490231 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f669c2_c88e_48b1_9b6d_c84b44878a95.slice/crio-4139ac2c5a018d95f1986111f5b4f8013c3155dbff95fc38954b55278784423b WatchSource:0}: Error finding container 4139ac2c5a018d95f1986111f5b4f8013c3155dbff95fc38954b55278784423b: Status 404 returned error can't find the container with id 4139ac2c5a018d95f1986111f5b4f8013c3155dbff95fc38954b55278784423b Jul 28 14:24:33.020337 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:33.020298 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:33.113484 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:33.113441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:33.466730 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:33.466702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" event={"ID":"14f669c2-c88e-48b1-9b6d-c84b44878a95","Type":"ContainerStarted","Data":"4139ac2c5a018d95f1986111f5b4f8013c3155dbff95fc38954b55278784423b"} Jul 28 14:24:37.250566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.250528 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:24:37.250846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.250600 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:24:37.250846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.250631 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:24:37.273579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.273553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:37.456956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.456923 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-br66k"] Jul 28 14:24:37.471856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.471834 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l"] Jul 28 14:24:37.471986 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.471922 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:37.478043 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.478018 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 28 14:24:37.479530 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.479508 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp"] Jul 28 14:24:37.490653 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.490622 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bkxkj\"" Jul 28 14:24:37.493539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.493363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-br66k"] Jul 28 14:24:37.493539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.493418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" event={"ID":"14f669c2-c88e-48b1-9b6d-c84b44878a95","Type":"ContainerStarted","Data":"a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c"} Jul 28 14:24:37.493539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.493523 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.493955 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.493916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:37.497195 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.497174 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 28 14:24:37.497723 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.497706 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mz2hd\"" Jul 28 14:24:37.521264 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.521242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp"] Jul 28 14:24:37.591207 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.591184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgs7\" (UniqueName: \"kubernetes.io/projected/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-kube-api-access-7tgs7\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:37.591366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.591239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-cert\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:37.591366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.591266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82933eb6-2f55-41b8-9829-099ea0f50185-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7mgvp\" (UID: \"82933eb6-2f55-41b8-9829-099ea0f50185\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.591366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.591332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7clm\" (UniqueName: \"kubernetes.io/projected/82933eb6-2f55-41b8-9829-099ea0f50185-kube-api-access-s7clm\") pod \"llmisvc-controller-manager-777c6bd7d5-7mgvp\" (UID: \"82933eb6-2f55-41b8-9829-099ea0f50185\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.591536 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.591431 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" podStartSLOduration=1.833660155 podStartE2EDuration="6.59141538s" totalImagesPullingTime="4.757755225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:37.588183434 +0000 UTC m=+379.253425258" watchObservedRunningTime="2026-07-28 14:24:37.59141538 +0000 UTC m=+379.256657203" Jul 28 14:24:37.632695 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.632648 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:37.691932 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.691911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgs7\" (UniqueName: \"kubernetes.io/projected/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-kube-api-access-7tgs7\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:37.692051 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.691968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-cert\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:37.692051 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.691991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82933eb6-2f55-41b8-9829-099ea0f50185-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7mgvp\" (UID: \"82933eb6-2f55-41b8-9829-099ea0f50185\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.692051 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.692021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7clm\" (UniqueName: \"kubernetes.io/projected/82933eb6-2f55-41b8-9829-099ea0f50185-kube-api-access-s7clm\") pod \"llmisvc-controller-manager-777c6bd7d5-7mgvp\" (UID: \"82933eb6-2f55-41b8-9829-099ea0f50185\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.692215 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:24:37.692060 2562 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 28 14:24:37.692215 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:24:37.692134 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-cert podName:ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:38.192113344 +0000 UTC m=+379.857355148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-cert") pod "kserve-controller-manager-d6f4b65b8-br66k" (UID: "ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1") : secret "kserve-webhook-server-cert" not found Jul 28 14:24:37.694812 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.694778 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82933eb6-2f55-41b8-9829-099ea0f50185-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7mgvp\" (UID: \"82933eb6-2f55-41b8-9829-099ea0f50185\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.706830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.706810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7clm\" (UniqueName: \"kubernetes.io/projected/82933eb6-2f55-41b8-9829-099ea0f50185-kube-api-access-s7clm\") pod \"llmisvc-controller-manager-777c6bd7d5-7mgvp\" (UID: \"82933eb6-2f55-41b8-9829-099ea0f50185\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.706942 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.706876 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgs7\" (UniqueName: \"kubernetes.io/projected/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-kube-api-access-7tgs7\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:37.803461 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.803442 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:37.952888 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.952855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:37.969351 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:37.969327 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp"] Jul 28 14:24:37.971222 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:24:37.971198 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82933eb6_2f55_41b8_9829_099ea0f50185.slice/crio-7c37e264369ed1f199195dda6c368b0ac088c71bdf5489aa4f6a4824bd564512 WatchSource:0}: Error finding container 7c37e264369ed1f199195dda6c368b0ac088c71bdf5489aa4f6a4824bd564512: Status 404 returned error can't find the container with id 7c37e264369ed1f199195dda6c368b0ac088c71bdf5489aa4f6a4824bd564512 Jul 28 14:24:38.195865 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.195805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-cert\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:38.197955 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.197932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1-cert\") pod \"kserve-controller-manager-d6f4b65b8-br66k\" (UID: \"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:38.306688 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.306654 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:38.307551 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.307531 2562 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 28 14:24:38.307603 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.307576 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" podUID="14f669c2-c88e-48b1-9b6d-c84b44878a95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 28 14:24:38.382751 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.382728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:38.497579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.497523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" event={"ID":"82933eb6-2f55-41b8-9829-099ea0f50185","Type":"ContainerStarted","Data":"7c37e264369ed1f199195dda6c368b0ac088c71bdf5489aa4f6a4824bd564512"} Jul 28 14:24:38.497750 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.497577 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" podUID="14f669c2-c88e-48b1-9b6d-c84b44878a95" containerName="istio-proxy" containerID="cri-o://a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c" gracePeriod=30 Jul 28 14:24:38.591746 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.591708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-br66k"] Jul 28 14:24:38.595092 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:24:38.595060 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2e92d3_7ea2_4ab6_a00f_ce103e4b1bf1.slice/crio-4040ca4f352361f6593606d49223bd9cbbbc7c48e8b4f0536884f6004e2c5616 WatchSource:0}: Error finding container 4040ca4f352361f6593606d49223bd9cbbbc7c48e8b4f0536884f6004e2c5616: Status 404 returned error can't find the container with id 4040ca4f352361f6593606d49223bd9cbbbc7c48e8b4f0536884f6004e2c5616 Jul 28 14:24:38.772408 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.772336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:38.874187 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:38.874150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:39.147154 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:39.147128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:24:39.508774 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:39.508639 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" event={"ID":"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1","Type":"ContainerStarted","Data":"4040ca4f352361f6593606d49223bd9cbbbc7c48e8b4f0536884f6004e2c5616"} Jul 28 14:24:42.525207 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:42.525168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" event={"ID":"ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1","Type":"ContainerStarted","Data":"baab7b18216c76439d3433eda94e8f3b927b89a3ea300599036e96e91c0907ac"} Jul 28 14:24:42.525669 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:42.525330 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:24:42.526648 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:42.526626 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" event={"ID":"82933eb6-2f55-41b8-9829-099ea0f50185","Type":"ContainerStarted","Data":"e8977a115092337da517ba3645d0546f5fd9e7f80710f3e789dcfe52d91e63df"} Jul 28 14:24:42.526757 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:42.526743 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:24:42.546699 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:42.546645 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" podStartSLOduration=2.662961527 podStartE2EDuration="5.546631072s" totalImagesPullingTime="2.883669545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:42.544232454 +0000 UTC m=+384.209474276" watchObservedRunningTime="2026-07-28 14:24:42.546631072 +0000 UTC m=+384.211872895" Jul 28 14:24:43.763323 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.763265 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:43.787662 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.787615 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" podStartSLOduration=3.2757261140000002 podStartE2EDuration="6.787599333s" totalImagesPullingTime="3.511873219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:42.567524971 +0000 UTC m=+384.232766794" watchObservedRunningTime="2026-07-28 14:24:43.787599333 +0000 UTC m=+385.452841167" Jul 28 14:24:43.841725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841700 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-token\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.841851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841737 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-data\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.841851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841773 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-envoy\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.841851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841789 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-socket\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.841851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841822 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfkv8\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-kube-api-access-lfkv8\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.841851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841847 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-credential-socket\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.842114 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841884 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f669c2-c88e-48b1-9b6d-c84b44878a95-istiod-ca-cert\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.842114 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841976 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-certs\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.842114 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.841980 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-data" (OuterVolumeSpecName: "istio-data") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:24:43.842114 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842011 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-podinfo\") pod \"14f669c2-c88e-48b1-9b6d-c84b44878a95\" (UID: \"14f669c2-c88e-48b1-9b6d-c84b44878a95\") " Jul 28 14:24:43.842307 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842154 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:24:43.842307 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842249 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:24:43.842307 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842286 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:24:43.842451 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f669c2-c88e-48b1-9b6d-c84b44878a95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:24:43.842451 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842352 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.842451 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842371 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.842451 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842386 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.842451 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.842399 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.844108 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.844078 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:24:43.844206 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.844114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 14:24:43.844419 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.844392 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-token" (OuterVolumeSpecName: "istio-token") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:24:43.844419 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.844395 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-kube-api-access-lfkv8" (OuterVolumeSpecName: "kube-api-access-lfkv8") pod "14f669c2-c88e-48b1-9b6d-c84b44878a95" (UID: "14f669c2-c88e-48b1-9b6d-c84b44878a95"). InnerVolumeSpecName "kube-api-access-lfkv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:24:43.943038 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.943017 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.943038 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.943036 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkv8\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-kube-api-access-lfkv8\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.943155 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.943046 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14f669c2-c88e-48b1-9b6d-c84b44878a95-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.943155 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.943055 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:43.943155 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:43.943063 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14f669c2-c88e-48b1-9b6d-c84b44878a95-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:24:44.537518 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.537478 2562 generic.go:356] "Generic (PLEG): container finished" podID="14f669c2-c88e-48b1-9b6d-c84b44878a95" containerID="a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c" exitCode=0 Jul 28 14:24:44.537666 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.537566 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" Jul 28 14:24:44.537666 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.537560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" event={"ID":"14f669c2-c88e-48b1-9b6d-c84b44878a95","Type":"ContainerDied","Data":"a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c"} Jul 28 14:24:44.537666 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.537628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l" event={"ID":"14f669c2-c88e-48b1-9b6d-c84b44878a95","Type":"ContainerDied","Data":"4139ac2c5a018d95f1986111f5b4f8013c3155dbff95fc38954b55278784423b"} Jul 28 14:24:44.537666 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.537655 2562 scope.go:117] "RemoveContainer" containerID="a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c" Jul 28 14:24:44.547306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.547284 2562 scope.go:117] "RemoveContainer" containerID="a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c" Jul 28 14:24:44.547577 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:24:44.547557 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c\": container with ID starting with a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c not found: ID does not exist" containerID="a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c" Jul 28 14:24:44.547647 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.547589 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c"} err="failed to get container status \"a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c\": rpc error: code = NotFound desc = could not find container \"a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c\": container with ID starting with a5fbcd60ffd43d2b2f36918fdeddb82e9379aa8cc4f7ae81ca71a59c6cfbc34c not found: ID does not exist" Jul 28 14:24:44.559946 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.559919 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l"] Jul 28 14:24:44.564401 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.564377 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c4cf4685fjhg4l"] Jul 28 14:24:44.888347 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:44.888322 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f669c2-c88e-48b1-9b6d-c84b44878a95" path="/var/lib/kubelet/pods/14f669c2-c88e-48b1-9b6d-c84b44878a95/volumes" Jul 28 14:24:45.264004 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:24:45.263931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:13.533523 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:13.533479 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7mgvp" Jul 28 14:25:13.537216 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:13.537182 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-br66k" Jul 28 14:25:25.315041 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:25.315005 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:25.436185 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:25.436146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:25.619895 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:25.619806 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:40.471259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.471227 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcb87dbb9-9bl5s"] Jul 28 14:25:40.471727 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.471707 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f669c2-c88e-48b1-9b6d-c84b44878a95" containerName="istio-proxy" Jul 28 14:25:40.471780 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.471731 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f669c2-c88e-48b1-9b6d-c84b44878a95" containerName="istio-proxy" Jul 28 14:25:40.471839 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.471828 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f669c2-c88e-48b1-9b6d-c84b44878a95" containerName="istio-proxy" Jul 28 14:25:40.475260 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.475239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.500878 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.500848 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcb87dbb9-9bl5s"] Jul 28 14:25:40.553207 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.553177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:40.568288 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-trusted-ca-bundle\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.568436 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-oauth-serving-cert\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.568436 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568348 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-oauth-config\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.568436 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-config\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.568585 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-serving-cert\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.568585 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-service-ca\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.568585 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.568520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnbs\" (UniqueName: \"kubernetes.io/projected/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-kube-api-access-ddnbs\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669610 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669575 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-oauth-config\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-config\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-serving-cert\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-service-ca\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnbs\" (UniqueName: \"kubernetes.io/projected/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-kube-api-access-ddnbs\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-trusted-ca-bundle\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.669849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.669832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-oauth-serving-cert\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.670516 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.670488 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-config\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.670640 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.670497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-service-ca\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.670736 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.670568 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-oauth-serving-cert\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.670867 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.670843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-trusted-ca-bundle\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.672273 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.672246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-oauth-config\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.672379 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.672362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-console-serving-cert\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.681457 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.681421 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnbs\" (UniqueName: \"kubernetes.io/projected/c22c4c7e-c5f5-4ab7-a402-c7a16b550525-kube-api-access-ddnbs\") pod \"console-5bcb87dbb9-9bl5s\" (UID: \"c22c4c7e-c5f5-4ab7-a402-c7a16b550525\") " pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.784712 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.784593 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:40.930307 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:40.930265 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcb87dbb9-9bl5s"] Jul 28 14:25:40.932567 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:25:40.932541 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22c4c7e_c5f5_4ab7_a402_c7a16b550525.slice/crio-64fbb019a2076486f098f328c1333e885fce51ab512ad54a7171ec00b14f2770 WatchSource:0}: Error finding container 64fbb019a2076486f098f328c1333e885fce51ab512ad54a7171ec00b14f2770: Status 404 returned error can't find the container with id 64fbb019a2076486f098f328c1333e885fce51ab512ad54a7171ec00b14f2770 Jul 28 14:25:41.211073 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.211034 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt"] Jul 28 14:25:41.213778 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.213761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.216423 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.216394 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 14:25:41.216964 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.216930 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 14:25:41.216964 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.216951 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-28hpc\"" Jul 28 14:25:41.226664 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.226634 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt"] Jul 28 14:25:41.273572 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.273533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnr6x\" (UniqueName: \"kubernetes.io/projected/da195d31-8d76-4240-b6d0-929aeaac932d-kube-api-access-hnr6x\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.273572 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.273579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.273824 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.273604 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.374478 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.374445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnr6x\" (UniqueName: \"kubernetes.io/projected/da195d31-8d76-4240-b6d0-929aeaac932d-kube-api-access-hnr6x\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.374478 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.374485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.374700 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.374510 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.374923 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.374901 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.374970 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.374912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.383998 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.383976 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnr6x\" (UniqueName: \"kubernetes.io/projected/da195d31-8d76-4240-b6d0-929aeaac932d-kube-api-access-hnr6x\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.523856 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.523767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:41.615213 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.615184 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8"] Jul 28 14:25:41.619953 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.619932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.634799 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.634773 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8"] Jul 28 14:25:41.662366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.662343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt"] Jul 28 14:25:41.664308 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:25:41.664282 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda195d31_8d76_4240_b6d0_929aeaac932d.slice/crio-0e57fe5d6408b9ddab84d58ee671c698ae6e03aa16f699e0778790d814ef799a WatchSource:0}: Error finding container 0e57fe5d6408b9ddab84d58ee671c698ae6e03aa16f699e0778790d814ef799a: Status 404 returned error can't find the container with id 0e57fe5d6408b9ddab84d58ee671c698ae6e03aa16f699e0778790d814ef799a Jul 28 14:25:41.677026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.677003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tv7\" (UniqueName: \"kubernetes.io/projected/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-kube-api-access-b5tv7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.677111 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.677047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.677160 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.677137 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.778019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.777985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.778177 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.778031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.778177 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.778100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tv7\" (UniqueName: \"kubernetes.io/projected/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-kube-api-access-b5tv7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.778435 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.778413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.778494 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.778421 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.787941 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.787922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tv7\" (UniqueName: \"kubernetes.io/projected/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-kube-api-access-b5tv7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:41.819192 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.819168 2562 generic.go:356] "Generic (PLEG): container finished" podID="da195d31-8d76-4240-b6d0-929aeaac932d" containerID="6d691c2ba9f4f614d1833d6e81754d322b17198ede131ad43b9d5a8514e7042d" exitCode=0 Jul 28 14:25:41.819304 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.819257 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" event={"ID":"da195d31-8d76-4240-b6d0-929aeaac932d","Type":"ContainerDied","Data":"6d691c2ba9f4f614d1833d6e81754d322b17198ede131ad43b9d5a8514e7042d"} Jul 28 14:25:41.819363 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.819308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" event={"ID":"da195d31-8d76-4240-b6d0-929aeaac932d","Type":"ContainerStarted","Data":"0e57fe5d6408b9ddab84d58ee671c698ae6e03aa16f699e0778790d814ef799a"} Jul 28 14:25:41.821046 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.821023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb87dbb9-9bl5s" event={"ID":"c22c4c7e-c5f5-4ab7-a402-c7a16b550525","Type":"ContainerStarted","Data":"2adb05faf18285f4491297d85775862c5d71cf8bb29b7a2759ca3e8e0eaf0855"} Jul 28 14:25:41.821133 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.821068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcb87dbb9-9bl5s" event={"ID":"c22c4c7e-c5f5-4ab7-a402-c7a16b550525","Type":"ContainerStarted","Data":"64fbb019a2076486f098f328c1333e885fce51ab512ad54a7171ec00b14f2770"} Jul 28 14:25:41.859476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.859399 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bcb87dbb9-9bl5s" podStartSLOduration=1.859382193 podStartE2EDuration="1.859382193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:41.858256224 +0000 UTC m=+443.523498047" watchObservedRunningTime="2026-07-28 14:25:41.859382193 +0000 UTC m=+443.524624080" Jul 28 14:25:41.931228 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:41.931194 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:42.001876 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.001843 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q"] Jul 28 14:25:42.005743 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.005701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.013283 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.013250 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q"] Jul 28 14:25:42.074944 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.074921 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8"] Jul 28 14:25:42.076125 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:25:42.076099 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8259e71_b3f1_4ce3_bc6a_1708759f8df1.slice/crio-27faf58ac054de773bc29ffa23853c1d631ce8e18b880255586a222f85fa5843 WatchSource:0}: Error finding container 27faf58ac054de773bc29ffa23853c1d631ce8e18b880255586a222f85fa5843: Status 404 returned error can't find the container with id 27faf58ac054de773bc29ffa23853c1d631ce8e18b880255586a222f85fa5843 Jul 28 14:25:42.081230 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.081204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.081350 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.081334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.081426 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.081378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgjg\" (UniqueName: \"kubernetes.io/projected/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-kube-api-access-tfgjg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.182748 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.182702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.182869 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.182765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgjg\" (UniqueName: \"kubernetes.io/projected/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-kube-api-access-tfgjg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.182869 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.182796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.182993 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.182966 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.183064 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.183046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.192853 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.192830 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgjg\" (UniqueName: \"kubernetes.io/projected/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-kube-api-access-tfgjg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.317061 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.317027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:42.452349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.452326 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q"] Jul 28 14:25:42.454247 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:25:42.454216 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cd38f1_a086_4342_9f30_5115bc0b4f5e.slice/crio-446e85d3ec7a2feffb389d3f09ee5a702e78ab8c1e48ffde627955cab853413d WatchSource:0}: Error finding container 446e85d3ec7a2feffb389d3f09ee5a702e78ab8c1e48ffde627955cab853413d: Status 404 returned error can't find the container with id 446e85d3ec7a2feffb389d3f09ee5a702e78ab8c1e48ffde627955cab853413d Jul 28 14:25:42.607973 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.607895 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6"] Jul 28 14:25:42.611790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.611771 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.622054 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.622029 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6"] Jul 28 14:25:42.687818 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.687790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.687960 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.687858 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.687960 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.687908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gkc\" (UniqueName: \"kubernetes.io/projected/db07b740-8d8a-4129-87c2-b65d291b8c2b-kube-api-access-t2gkc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.789291 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.789249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.789491 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.789297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gkc\" (UniqueName: \"kubernetes.io/projected/db07b740-8d8a-4129-87c2-b65d291b8c2b-kube-api-access-t2gkc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.789491 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.789365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.789768 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.789744 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.789826 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.789754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.798772 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.798746 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gkc\" (UniqueName: \"kubernetes.io/projected/db07b740-8d8a-4129-87c2-b65d291b8c2b-kube-api-access-t2gkc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:42.832236 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.832192 2562 generic.go:356] "Generic (PLEG): container finished" podID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerID="58798a79c24ac4495487d2c457d7fe73feb21061002ba40b9a883e6c3239d2e2" exitCode=0 Jul 28 14:25:42.832416 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.832238 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" event={"ID":"f8259e71-b3f1-4ce3-bc6a-1708759f8df1","Type":"ContainerDied","Data":"58798a79c24ac4495487d2c457d7fe73feb21061002ba40b9a883e6c3239d2e2"} Jul 28 14:25:42.832416 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.832323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" event={"ID":"f8259e71-b3f1-4ce3-bc6a-1708759f8df1","Type":"ContainerStarted","Data":"27faf58ac054de773bc29ffa23853c1d631ce8e18b880255586a222f85fa5843"} Jul 28 14:25:42.833971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.833947 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerID="4e8b325534dad4187326bc923d29c4528f0293e1660743c9d8bf53ce5de616de" exitCode=0 Jul 28 14:25:42.834110 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.834031 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" event={"ID":"d4cd38f1-a086-4342-9f30-5115bc0b4f5e","Type":"ContainerDied","Data":"4e8b325534dad4187326bc923d29c4528f0293e1660743c9d8bf53ce5de616de"} Jul 28 14:25:42.834110 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.834068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" event={"ID":"d4cd38f1-a086-4342-9f30-5115bc0b4f5e","Type":"ContainerStarted","Data":"446e85d3ec7a2feffb389d3f09ee5a702e78ab8c1e48ffde627955cab853413d"} Jul 28 14:25:42.835965 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.835943 2562 generic.go:356] "Generic (PLEG): container finished" podID="da195d31-8d76-4240-b6d0-929aeaac932d" containerID="8f5de151005ea4a4ac6cfcc169fbe4a42911a24ba6dbd117673317dea0210383" exitCode=0 Jul 28 14:25:42.836115 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.836044 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" event={"ID":"da195d31-8d76-4240-b6d0-929aeaac932d","Type":"ContainerDied","Data":"8f5de151005ea4a4ac6cfcc169fbe4a42911a24ba6dbd117673317dea0210383"} Jul 28 14:25:42.929199 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:42.929167 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:43.072559 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.072529 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6"] Jul 28 14:25:43.074443 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:25:43.074419 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb07b740_8d8a_4129_87c2_b65d291b8c2b.slice/crio-0a416ef77d4578e8686680123ad3d9d39cc88446043dcb263925ab8123f9605a WatchSource:0}: Error finding container 0a416ef77d4578e8686680123ad3d9d39cc88446043dcb263925ab8123f9605a: Status 404 returned error can't find the container with id 0a416ef77d4578e8686680123ad3d9d39cc88446043dcb263925ab8123f9605a Jul 28 14:25:43.842306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.842275 2562 generic.go:356] "Generic (PLEG): container finished" podID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerID="e8f0bfd4ccbd7eeb266a325639f33a699b41799ef29b160d7982bba6cc05cf6a" exitCode=0 Jul 28 14:25:43.842670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.842348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" event={"ID":"db07b740-8d8a-4129-87c2-b65d291b8c2b","Type":"ContainerDied","Data":"e8f0bfd4ccbd7eeb266a325639f33a699b41799ef29b160d7982bba6cc05cf6a"} Jul 28 14:25:43.842670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.842397 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" event={"ID":"db07b740-8d8a-4129-87c2-b65d291b8c2b","Type":"ContainerStarted","Data":"0a416ef77d4578e8686680123ad3d9d39cc88446043dcb263925ab8123f9605a"} Jul 28 14:25:43.844697 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.844647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" event={"ID":"f8259e71-b3f1-4ce3-bc6a-1708759f8df1","Type":"ContainerStarted","Data":"920a5218da4311aec3e7b2f6ab1735eba4252e9a2248770f1227dfc0486194cc"} Jul 28 14:25:43.847024 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.846992 2562 generic.go:356] "Generic (PLEG): container finished" podID="da195d31-8d76-4240-b6d0-929aeaac932d" containerID="bd008b75418c6022d332822f9c335dd00ec2620fc45bf9dee8d671b4a84cd2c7" exitCode=0 Jul 28 14:25:43.847118 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:43.847023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" event={"ID":"da195d31-8d76-4240-b6d0-929aeaac932d","Type":"ContainerDied","Data":"bd008b75418c6022d332822f9c335dd00ec2620fc45bf9dee8d671b4a84cd2c7"} Jul 28 14:25:44.853916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:44.853885 2562 generic.go:356] "Generic (PLEG): container finished" podID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerID="e9101b9cf6e5ada5fd570a5a846353b2d492c5ceeff6cf874a3e9df21fcd0c0d" exitCode=0 Jul 28 14:25:44.854346 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:44.853974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" event={"ID":"db07b740-8d8a-4129-87c2-b65d291b8c2b","Type":"ContainerDied","Data":"e9101b9cf6e5ada5fd570a5a846353b2d492c5ceeff6cf874a3e9df21fcd0c0d"} Jul 28 14:25:44.855941 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:44.855921 2562 generic.go:356] "Generic (PLEG): container finished" podID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerID="920a5218da4311aec3e7b2f6ab1735eba4252e9a2248770f1227dfc0486194cc" exitCode=0 Jul 28 14:25:44.856040 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:44.856006 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" event={"ID":"f8259e71-b3f1-4ce3-bc6a-1708759f8df1","Type":"ContainerDied","Data":"920a5218da4311aec3e7b2f6ab1735eba4252e9a2248770f1227dfc0486194cc"} Jul 28 14:25:44.857799 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:44.857779 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerID="ef5205b29270134e415fb670086608cd2514e6ed4ba83de09b815fcbac0a4bc0" exitCode=0 Jul 28 14:25:44.857911 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:44.857857 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" event={"ID":"d4cd38f1-a086-4342-9f30-5115bc0b4f5e","Type":"ContainerDied","Data":"ef5205b29270134e415fb670086608cd2514e6ed4ba83de09b815fcbac0a4bc0"} Jul 28 14:25:45.020073 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.019776 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:45.111892 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.111853 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-bundle\") pod \"da195d31-8d76-4240-b6d0-929aeaac932d\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " Jul 28 14:25:45.112054 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.111901 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-util\") pod \"da195d31-8d76-4240-b6d0-929aeaac932d\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " Jul 28 14:25:45.112054 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.111950 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnr6x\" (UniqueName: \"kubernetes.io/projected/da195d31-8d76-4240-b6d0-929aeaac932d-kube-api-access-hnr6x\") pod \"da195d31-8d76-4240-b6d0-929aeaac932d\" (UID: \"da195d31-8d76-4240-b6d0-929aeaac932d\") " Jul 28 14:25:45.112456 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.112426 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-bundle" (OuterVolumeSpecName: "bundle") pod "da195d31-8d76-4240-b6d0-929aeaac932d" (UID: "da195d31-8d76-4240-b6d0-929aeaac932d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:45.114305 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.114280 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da195d31-8d76-4240-b6d0-929aeaac932d-kube-api-access-hnr6x" (OuterVolumeSpecName: "kube-api-access-hnr6x") pod "da195d31-8d76-4240-b6d0-929aeaac932d" (UID: "da195d31-8d76-4240-b6d0-929aeaac932d"). InnerVolumeSpecName "kube-api-access-hnr6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:25:45.117749 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.117719 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-util" (OuterVolumeSpecName: "util") pod "da195d31-8d76-4240-b6d0-929aeaac932d" (UID: "da195d31-8d76-4240-b6d0-929aeaac932d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:45.213375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.213339 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:45.213375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.213371 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnr6x\" (UniqueName: \"kubernetes.io/projected/da195d31-8d76-4240-b6d0-929aeaac932d-kube-api-access-hnr6x\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:45.213375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.213381 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da195d31-8d76-4240-b6d0-929aeaac932d-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:45.865063 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.865036 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" Jul 28 14:25:45.865522 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.865032 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt" event={"ID":"da195d31-8d76-4240-b6d0-929aeaac932d","Type":"ContainerDied","Data":"0e57fe5d6408b9ddab84d58ee671c698ae6e03aa16f699e0778790d814ef799a"} Jul 28 14:25:45.865522 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.865194 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e57fe5d6408b9ddab84d58ee671c698ae6e03aa16f699e0778790d814ef799a" Jul 28 14:25:45.867105 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.867084 2562 generic.go:356] "Generic (PLEG): container finished" podID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerID="7ad9adabd80f9b402e75c4670403dd2893c5f6b1b280e187bbd50b123af835b3" exitCode=0 Jul 28 14:25:45.867227 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.867176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" event={"ID":"db07b740-8d8a-4129-87c2-b65d291b8c2b","Type":"ContainerDied","Data":"7ad9adabd80f9b402e75c4670403dd2893c5f6b1b280e187bbd50b123af835b3"} Jul 28 14:25:45.869278 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.869257 2562 generic.go:356] "Generic (PLEG): container finished" podID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerID="88530308de9d8fcb62080bfda0e7fa0038be99f805925355905c3fb1ab1b0266" exitCode=0 Jul 28 14:25:45.869368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.869341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" event={"ID":"f8259e71-b3f1-4ce3-bc6a-1708759f8df1","Type":"ContainerDied","Data":"88530308de9d8fcb62080bfda0e7fa0038be99f805925355905c3fb1ab1b0266"} Jul 28 14:25:45.871307 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.871283 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerID="f81259df383d8fe7158922670d90ceeed83127f5bd24257f863abbdefc64876c" exitCode=0 Jul 28 14:25:45.871412 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:45.871326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" event={"ID":"d4cd38f1-a086-4342-9f30-5115bc0b4f5e","Type":"ContainerDied","Data":"f81259df383d8fe7158922670d90ceeed83127f5bd24257f863abbdefc64876c"} Jul 28 14:25:47.021926 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.021901 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:47.061816 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.061785 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:47.064915 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.064891 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:47.132141 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132056 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-bundle\") pod \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " Jul 28 14:25:47.132141 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132105 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-util\") pod \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfgjg\" (UniqueName: \"kubernetes.io/projected/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-kube-api-access-tfgjg\") pod \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-bundle\") pod \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\" (UID: \"d4cd38f1-a086-4342-9f30-5115bc0b4f5e\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132204 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2gkc\" (UniqueName: \"kubernetes.io/projected/db07b740-8d8a-4129-87c2-b65d291b8c2b-kube-api-access-t2gkc\") pod \"db07b740-8d8a-4129-87c2-b65d291b8c2b\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132224 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-util\") pod \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132240 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-bundle\") pod \"db07b740-8d8a-4129-87c2-b65d291b8c2b\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132263 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-util\") pod \"db07b740-8d8a-4129-87c2-b65d291b8c2b\" (UID: \"db07b740-8d8a-4129-87c2-b65d291b8c2b\") " Jul 28 14:25:47.132368 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132327 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5tv7\" (UniqueName: \"kubernetes.io/projected/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-kube-api-access-b5tv7\") pod \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\" (UID: \"f8259e71-b3f1-4ce3-bc6a-1708759f8df1\") " Jul 28 14:25:47.132903 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.132840 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-bundle" (OuterVolumeSpecName: "bundle") pod "f8259e71-b3f1-4ce3-bc6a-1708759f8df1" (UID: "f8259e71-b3f1-4ce3-bc6a-1708759f8df1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:47.133178 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.133138 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-bundle" (OuterVolumeSpecName: "bundle") pod "d4cd38f1-a086-4342-9f30-5115bc0b4f5e" (UID: "d4cd38f1-a086-4342-9f30-5115bc0b4f5e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:47.133776 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.133753 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-bundle" (OuterVolumeSpecName: "bundle") pod "db07b740-8d8a-4129-87c2-b65d291b8c2b" (UID: "db07b740-8d8a-4129-87c2-b65d291b8c2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:47.134888 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.134862 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db07b740-8d8a-4129-87c2-b65d291b8c2b-kube-api-access-t2gkc" (OuterVolumeSpecName: "kube-api-access-t2gkc") pod "db07b740-8d8a-4129-87c2-b65d291b8c2b" (UID: "db07b740-8d8a-4129-87c2-b65d291b8c2b"). InnerVolumeSpecName "kube-api-access-t2gkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:25:47.134888 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.134897 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-kube-api-access-tfgjg" (OuterVolumeSpecName: "kube-api-access-tfgjg") pod "d4cd38f1-a086-4342-9f30-5115bc0b4f5e" (UID: "d4cd38f1-a086-4342-9f30-5115bc0b4f5e"). InnerVolumeSpecName "kube-api-access-tfgjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:25:47.135090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.135074 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-kube-api-access-b5tv7" (OuterVolumeSpecName: "kube-api-access-b5tv7") pod "f8259e71-b3f1-4ce3-bc6a-1708759f8df1" (UID: "f8259e71-b3f1-4ce3-bc6a-1708759f8df1"). InnerVolumeSpecName "kube-api-access-b5tv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:25:47.138273 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.138239 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-util" (OuterVolumeSpecName: "util") pod "d4cd38f1-a086-4342-9f30-5115bc0b4f5e" (UID: "d4cd38f1-a086-4342-9f30-5115bc0b4f5e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:47.138987 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.138966 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-util" (OuterVolumeSpecName: "util") pod "f8259e71-b3f1-4ce3-bc6a-1708759f8df1" (UID: "f8259e71-b3f1-4ce3-bc6a-1708759f8df1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:47.140062 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.140038 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-util" (OuterVolumeSpecName: "util") pod "db07b740-8d8a-4129-87c2-b65d291b8c2b" (UID: "db07b740-8d8a-4129-87c2-b65d291b8c2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:25:47.234011 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.233961 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfgjg\" (UniqueName: \"kubernetes.io/projected/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-kube-api-access-tfgjg\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234011 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234004 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234011 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234015 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2gkc\" (UniqueName: \"kubernetes.io/projected/db07b740-8d8a-4129-87c2-b65d291b8c2b-kube-api-access-t2gkc\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234011 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234025 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234011 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234033 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234041 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db07b740-8d8a-4129-87c2-b65d291b8c2b-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234049 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5tv7\" (UniqueName: \"kubernetes.io/projected/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-kube-api-access-b5tv7\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234057 2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8259e71-b3f1-4ce3-bc6a-1708759f8df1-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.234348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.234064 2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d4cd38f1-a086-4342-9f30-5115bc0b4f5e-util\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:25:47.889351 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.889320 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" Jul 28 14:25:47.889533 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.889350 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6" event={"ID":"db07b740-8d8a-4129-87c2-b65d291b8c2b","Type":"ContainerDied","Data":"0a416ef77d4578e8686680123ad3d9d39cc88446043dcb263925ab8123f9605a"} Jul 28 14:25:47.889533 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.889392 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a416ef77d4578e8686680123ad3d9d39cc88446043dcb263925ab8123f9605a" Jul 28 14:25:47.891250 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.891230 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" Jul 28 14:25:47.891382 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.891254 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8" event={"ID":"f8259e71-b3f1-4ce3-bc6a-1708759f8df1","Type":"ContainerDied","Data":"27faf58ac054de773bc29ffa23853c1d631ce8e18b880255586a222f85fa5843"} Jul 28 14:25:47.891382 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.891293 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27faf58ac054de773bc29ffa23853c1d631ce8e18b880255586a222f85fa5843" Jul 28 14:25:47.893199 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.893176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" event={"ID":"d4cd38f1-a086-4342-9f30-5115bc0b4f5e","Type":"ContainerDied","Data":"446e85d3ec7a2feffb389d3f09ee5a702e78ab8c1e48ffde627955cab853413d"} Jul 28 14:25:47.893311 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.893208 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q" Jul 28 14:25:47.893311 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:47.893214 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="446e85d3ec7a2feffb389d3f09ee5a702e78ab8c1e48ffde627955cab853413d" Jul 28 14:25:50.785401 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:50.785361 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:50.785931 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:50.785515 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:50.791119 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:50.791094 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:50.911069 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:50.911040 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bcb87dbb9-9bl5s" Jul 28 14:25:50.969130 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:50.969097 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69b6f97ccc-nvs87"] Jul 28 14:25:52.261122 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:52.261087 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:53.368887 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:53.368841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:54.856762 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:54.856713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:25:59.206883 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:25:59.206845 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:01.200711 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.200664 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm"] Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201009 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="util" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201019 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="util" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201029 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="util" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201035 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="util" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201042 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="extract" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201049 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="extract" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201057 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="extract" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201062 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="extract" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201071 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="pull" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201076 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="pull" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201083 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="extract" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201088 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="extract" Jul 28 14:26:01.201090 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201094 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="util" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201099 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="util" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201107 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="pull" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201113 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="pull" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201123 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="extract" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201128 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="extract" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201134 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="pull" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201139 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="pull" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201143 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="util" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201148 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="util" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201156 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="pull" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201162 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="pull" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201212 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da195d31-8d76-4240-b6d0-929aeaac932d" containerName="extract" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201221 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4cd38f1-a086-4342-9f30-5115bc0b4f5e" containerName="extract" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201227 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="db07b740-8d8a-4129-87c2-b65d291b8c2b" containerName="extract" Jul 28 14:26:01.201733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.201232 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8259e71-b3f1-4ce3-bc6a-1708759f8df1" containerName="extract" Jul 28 14:26:01.205524 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.205502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:01.209294 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.209274 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xkrcr\"" Jul 28 14:26:01.209790 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.209771 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 14:26:01.209853 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.209818 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 14:26:01.209901 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.209827 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 28 14:26:01.218579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.218552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm"] Jul 28 14:26:01.260367 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.260337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnbs\" (UniqueName: \"kubernetes.io/projected/8798c72d-80b7-4216-ac70-3c02639da1e9-kube-api-access-wgnbs\") pod \"dns-operator-controller-manager-864dd78c4c-54dfm\" (UID: \"8798c72d-80b7-4216-ac70-3c02639da1e9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:01.361312 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.361273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnbs\" (UniqueName: \"kubernetes.io/projected/8798c72d-80b7-4216-ac70-3c02639da1e9-kube-api-access-wgnbs\") pod \"dns-operator-controller-manager-864dd78c4c-54dfm\" (UID: \"8798c72d-80b7-4216-ac70-3c02639da1e9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:01.372235 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.372204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnbs\" (UniqueName: \"kubernetes.io/projected/8798c72d-80b7-4216-ac70-3c02639da1e9-kube-api-access-wgnbs\") pod \"dns-operator-controller-manager-864dd78c4c-54dfm\" (UID: \"8798c72d-80b7-4216-ac70-3c02639da1e9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:01.516838 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.516739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:01.671242 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.671197 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm"] Jul 28 14:26:01.673587 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:26:01.673552 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8798c72d_80b7_4216_ac70_3c02639da1e9.slice/crio-172c14d226ac15d048dbafaa4d1f0ac9464d641801a95d005c8ea1713625802d WatchSource:0}: Error finding container 172c14d226ac15d048dbafaa4d1f0ac9464d641801a95d005c8ea1713625802d: Status 404 returned error can't find the container with id 172c14d226ac15d048dbafaa4d1f0ac9464d641801a95d005c8ea1713625802d Jul 28 14:26:01.948228 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.948191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:01.961137 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:01.961104 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" event={"ID":"8798c72d-80b7-4216-ac70-3c02639da1e9","Type":"ContainerStarted","Data":"172c14d226ac15d048dbafaa4d1f0ac9464d641801a95d005c8ea1713625802d"} Jul 28 14:26:03.276851 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:03.276813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:03.972905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:03.972821 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" event={"ID":"8798c72d-80b7-4216-ac70-3c02639da1e9","Type":"ContainerStarted","Data":"a29b68523b664b366d7d4c7a10835d69c442391a50a1ced077864474dcf83d5c"} Jul 28 14:26:03.973040 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:03.972925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:04.361055 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:04.361022 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:06.116261 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:06.116223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:07.124327 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:07.124291 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:07.780409 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:07.780378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:08.805106 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:08.805057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:10.661650 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.661593 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" podStartSLOduration=7.669671135 podStartE2EDuration="9.661573573s" totalImagesPullingTime="1.991902438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:26:04.001180446 +0000 UTC m=+465.666422270" watchObservedRunningTime="2026-07-28 14:26:10.661573573 +0000 UTC m=+472.326815395" Jul 28 14:26:10.662470 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.662452 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tztkf"] Jul 28 14:26:10.665936 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.665918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:10.684987 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.684966 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q9t29\"" Jul 28 14:26:10.700021 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.699991 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tztkf"] Jul 28 14:26:10.743436 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.743397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhrr6\" (UniqueName: \"kubernetes.io/projected/5e0522f1-d306-4ba2-82b9-386ff956491e-kube-api-access-dhrr6\") pod \"authorino-operator-75cc8d5cb7-tztkf\" (UID: \"5e0522f1-d306-4ba2-82b9-386ff956491e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:10.844087 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.844053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhrr6\" (UniqueName: \"kubernetes.io/projected/5e0522f1-d306-4ba2-82b9-386ff956491e-kube-api-access-dhrr6\") pod \"authorino-operator-75cc8d5cb7-tztkf\" (UID: \"5e0522f1-d306-4ba2-82b9-386ff956491e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:10.870490 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.870454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhrr6\" (UniqueName: \"kubernetes.io/projected/5e0522f1-d306-4ba2-82b9-386ff956491e-kube-api-access-dhrr6\") pod \"authorino-operator-75cc8d5cb7-tztkf\" (UID: \"5e0522f1-d306-4ba2-82b9-386ff956491e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:10.976516 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:10.976419 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:11.164415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:11.164388 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tztkf"] Jul 28 14:26:11.166214 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:26:11.166188 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0522f1_d306_4ba2_82b9_386ff956491e.slice/crio-98d96177f45de0d3d6a37549a4971b0f2ccf2af2fe06f1b3d825430647a24ef2 WatchSource:0}: Error finding container 98d96177f45de0d3d6a37549a4971b0f2ccf2af2fe06f1b3d825430647a24ef2: Status 404 returned error can't find the container with id 98d96177f45de0d3d6a37549a4971b0f2ccf2af2fe06f1b3d825430647a24ef2 Jul 28 14:26:11.907778 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:11.907745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:12.013318 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:12.013286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" event={"ID":"5e0522f1-d306-4ba2-82b9-386ff956491e","Type":"ContainerStarted","Data":"98d96177f45de0d3d6a37549a4971b0f2ccf2af2fe06f1b3d825430647a24ef2"} Jul 28 14:26:12.156929 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:12.156901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:14.026126 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:14.026056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" event={"ID":"5e0522f1-d306-4ba2-82b9-386ff956491e","Type":"ContainerStarted","Data":"4b1f9580510cb78b835bcd5d87252fab44c4e013e6e46d0dfd1ff547bdbe2865"} Jul 28 14:26:14.026440 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:14.026220 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:14.056527 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:14.056481 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" podStartSLOduration=1.4593044050000001 podStartE2EDuration="4.056464641s" totalImagesPullingTime="2.597160236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:26:14.055712209 +0000 UTC m=+475.720954035" watchObservedRunningTime="2026-07-28 14:26:14.056464641 +0000 UTC m=+475.721706464" Jul 28 14:26:14.582973 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:14.582935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:14.701375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:14.701337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:14.980339 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:14.980258 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-54dfm" Jul 28 14:26:15.990594 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:15.990551 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69b6f97ccc-nvs87" podUID="e2b70ceb-be47-4209-b150-673ec72fcba1" containerName="console" containerID="cri-o://6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667" gracePeriod=15 Jul 28 14:26:16.239355 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.239331 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:26:16.300014 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.299984 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-serving-cert\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300033 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-service-ca\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300057 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-trusted-ca-bundle\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbp6h\" (UniqueName: \"kubernetes.io/projected/e2b70ceb-be47-4209-b150-673ec72fcba1-kube-api-access-pbp6h\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300197 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-console-config\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300221 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-oauth-config\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300278 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-oauth-serving-cert\") pod \"e2b70ceb-be47-4209-b150-673ec72fcba1\" (UID: \"e2b70ceb-be47-4209-b150-673ec72fcba1\") " Jul 28 14:26:16.300501 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300427 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:26:16.300632 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-console-config" (OuterVolumeSpecName: "console-config") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:26:16.300792 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300621 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:26:16.300792 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.300744 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:26:16.302845 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.302813 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:26:16.302960 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.302836 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 14:26:16.302960 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.302877 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b70ceb-be47-4209-b150-673ec72fcba1-kube-api-access-pbp6h" (OuterVolumeSpecName: "kube-api-access-pbp6h") pod "e2b70ceb-be47-4209-b150-673ec72fcba1" (UID: "e2b70ceb-be47-4209-b150-673ec72fcba1"). InnerVolumeSpecName "kube-api-access-pbp6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:26:16.401770 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401738 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbp6h\" (UniqueName: \"kubernetes.io/projected/e2b70ceb-be47-4209-b150-673ec72fcba1-kube-api-access-pbp6h\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.401770 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401763 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-console-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.401770 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401772 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-oauth-config\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.401976 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401781 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-oauth-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.401976 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401790 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2b70ceb-be47-4209-b150-673ec72fcba1-console-serving-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.401976 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401798 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.401976 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.401807 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2b70ceb-be47-4209-b150-673ec72fcba1-trusted-ca-bundle\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:26:16.944935 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.944902 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-df78m"] Jul 28 14:26:16.945243 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.945231 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b70ceb-be47-4209-b150-673ec72fcba1" containerName="console" Jul 28 14:26:16.945287 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.945246 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b70ceb-be47-4209-b150-673ec72fcba1" containerName="console" Jul 28 14:26:16.945322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.945299 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b70ceb-be47-4209-b150-673ec72fcba1" containerName="console" Jul 28 14:26:16.949349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.949332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:16.952576 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.952537 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-28hpc\"" Jul 28 14:26:16.952796 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.952778 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 28 14:26:16.953237 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.953220 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 28 14:26:16.970435 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.970410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-df78m"] Jul 28 14:26:16.990841 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:16.990815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:17.007242 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.007217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fa1a911-90cb-491c-979c-140e3b2873d3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.007343 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.007264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fk2q\" (UniqueName: \"kubernetes.io/projected/3fa1a911-90cb-491c-979c-140e3b2873d3-kube-api-access-4fk2q\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.007343 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.007302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fa1a911-90cb-491c-979c-140e3b2873d3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.048937 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.048909 2562 generic.go:356] "Generic (PLEG): container finished" podID="e2b70ceb-be47-4209-b150-673ec72fcba1" containerID="6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667" exitCode=0 Jul 28 14:26:17.049030 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.048955 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69b6f97ccc-nvs87" Jul 28 14:26:17.049030 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.048992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b6f97ccc-nvs87" event={"ID":"e2b70ceb-be47-4209-b150-673ec72fcba1","Type":"ContainerDied","Data":"6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667"} Jul 28 14:26:17.049103 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.049041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69b6f97ccc-nvs87" event={"ID":"e2b70ceb-be47-4209-b150-673ec72fcba1","Type":"ContainerDied","Data":"bf91fd546711c6e2650a1e803f6708ca96acc5575b2087f4f3d8efc3baa9b51d"} Jul 28 14:26:17.049103 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.049062 2562 scope.go:117] "RemoveContainer" containerID="6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667" Jul 28 14:26:17.057588 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.057572 2562 scope.go:117] "RemoveContainer" containerID="6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667" Jul 28 14:26:17.057852 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:26:17.057833 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667\": container with ID starting with 6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667 not found: ID does not exist" containerID="6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667" Jul 28 14:26:17.057919 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.057862 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667"} err="failed to get container status \"6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667\": rpc error: code = NotFound desc = could not find container \"6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667\": container with ID starting with 6b25fb0cf8778caa551165dae6fc12ceafe756558b1623ac669f064d41c96667 not found: ID does not exist" Jul 28 14:26:17.108006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.107843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fk2q\" (UniqueName: \"kubernetes.io/projected/3fa1a911-90cb-491c-979c-140e3b2873d3-kube-api-access-4fk2q\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.108006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.107936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fa1a911-90cb-491c-979c-140e3b2873d3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.108211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.108010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fa1a911-90cb-491c-979c-140e3b2873d3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.108830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.108810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3fa1a911-90cb-491c-979c-140e3b2873d3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.110444 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.110427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fa1a911-90cb-491c-979c-140e3b2873d3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.111151 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.111132 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69b6f97ccc-nvs87"] Jul 28 14:26:17.126487 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.126465 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69b6f97ccc-nvs87"] Jul 28 14:26:17.133435 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.133407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fk2q\" (UniqueName: \"kubernetes.io/projected/3fa1a911-90cb-491c-979c-140e3b2873d3-kube-api-access-4fk2q\") pod \"kuadrant-console-plugin-66b5774985-df78m\" (UID: \"3fa1a911-90cb-491c-979c-140e3b2873d3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.258840 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.258762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" Jul 28 14:26:17.406131 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:17.406105 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-df78m"] Jul 28 14:26:17.408620 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:26:17.408586 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fa1a911_90cb_491c_979c_140e3b2873d3.slice/crio-72e2ae2d1b5d266999bab4a0efebf5993fab3428d920387cf957af2f10e8c875 WatchSource:0}: Error finding container 72e2ae2d1b5d266999bab4a0efebf5993fab3428d920387cf957af2f10e8c875: Status 404 returned error can't find the container with id 72e2ae2d1b5d266999bab4a0efebf5993fab3428d920387cf957af2f10e8c875 Jul 28 14:26:18.055263 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:18.055233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" event={"ID":"3fa1a911-90cb-491c-979c-140e3b2873d3","Type":"ContainerStarted","Data":"72e2ae2d1b5d266999bab4a0efebf5993fab3428d920387cf957af2f10e8c875"} Jul 28 14:26:18.490468 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:18.490335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:18.582033 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:18.581994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:18.890229 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:18.890202 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b70ceb-be47-4209-b150-673ec72fcba1" path="/var/lib/kubelet/pods/e2b70ceb-be47-4209-b150-673ec72fcba1/volumes" Jul 28 14:26:25.034774 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:25.034745 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tztkf" Jul 28 14:26:27.420151 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:27.420113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:27.682469 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:27.682382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:27.917532 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:27.917497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:30.108655 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:30.108607 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:30.235641 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:30.235563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:32.783547 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:32.783492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:33.006360 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:33.006323 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:37.010830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:37.010779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:37.222560 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:37.222522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:39.184405 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:39.184374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" event={"ID":"3fa1a911-90cb-491c-979c-140e3b2873d3","Type":"ContainerStarted","Data":"60cc29b612972ffcd17ecf281b6575d34e999cd9e70eab36450bc01cdc64e89d"} Jul 28 14:26:39.209738 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:39.209667 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-df78m" podStartSLOduration=1.616360316 podStartE2EDuration="23.209651488s" totalImagesPullingTime="21.593291172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:26:39.208232387 +0000 UTC m=+500.873474209" watchObservedRunningTime="2026-07-28 14:26:39.209651488 +0000 UTC m=+500.874893311" Jul 28 14:26:45.414249 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:45.414208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:45.602107 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:45.602075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:54.768826 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:54.768789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:26:54.870645 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:26:54.870614 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:00.123083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:00.123040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:00.480909 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:00.480831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:01.938120 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:01.938086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:02.055323 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.055293 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:02.137822 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.137789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:02.261748 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.261662 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:27:02.535784 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.535713 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:27:02.535784 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.535747 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:27:02.535959 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.535833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.539812 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.539786 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 14:27:02.600049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.600019 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa9894ff-4367-4136-b8c7-083562331b78-config-file\") pod \"limitador-limitador-54d6d988d5-6h9fn\" (UID: \"fa9894ff-4367-4136-b8c7-083562331b78\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.600204 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.600088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxmh\" (UniqueName: \"kubernetes.io/projected/fa9894ff-4367-4136-b8c7-083562331b78-kube-api-access-jkxmh\") pod \"limitador-limitador-54d6d988d5-6h9fn\" (UID: \"fa9894ff-4367-4136-b8c7-083562331b78\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.701315 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.701280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxmh\" (UniqueName: \"kubernetes.io/projected/fa9894ff-4367-4136-b8c7-083562331b78-kube-api-access-jkxmh\") pod \"limitador-limitador-54d6d988d5-6h9fn\" (UID: \"fa9894ff-4367-4136-b8c7-083562331b78\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.701487 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.701348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa9894ff-4367-4136-b8c7-083562331b78-config-file\") pod \"limitador-limitador-54d6d988d5-6h9fn\" (UID: \"fa9894ff-4367-4136-b8c7-083562331b78\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.701956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.701938 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa9894ff-4367-4136-b8c7-083562331b78-config-file\") pod \"limitador-limitador-54d6d988d5-6h9fn\" (UID: \"fa9894ff-4367-4136-b8c7-083562331b78\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.710596 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.710572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxmh\" (UniqueName: \"kubernetes.io/projected/fa9894ff-4367-4136-b8c7-083562331b78-kube-api-access-jkxmh\") pod \"limitador-limitador-54d6d988d5-6h9fn\" (UID: \"fa9894ff-4367-4136-b8c7-083562331b78\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.845659 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.845633 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:02.854650 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.854621 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:02.957098 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:02.957044 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:03.036281 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:03.036258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:27:03.038391 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:27:03.038362 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9894ff_4367_4136_b8c7_083562331b78.slice/crio-06b26e04f404bdc855a99c7e6dcbfaa25edc213334f3acac17ae4168cfc0584c WatchSource:0}: Error finding container 06b26e04f404bdc855a99c7e6dcbfaa25edc213334f3acac17ae4168cfc0584c: Status 404 returned error can't find the container with id 06b26e04f404bdc855a99c7e6dcbfaa25edc213334f3acac17ae4168cfc0584c Jul 28 14:27:03.305660 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:03.305630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" event={"ID":"fa9894ff-4367-4136-b8c7-083562331b78","Type":"ContainerStarted","Data":"06b26e04f404bdc855a99c7e6dcbfaa25edc213334f3acac17ae4168cfc0584c"} Jul 28 14:27:08.336519 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:08.336480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" event={"ID":"fa9894ff-4367-4136-b8c7-083562331b78","Type":"ContainerStarted","Data":"cedf277836f3549d2898c8c7399c88e4307f7590fe022c31e2494b07f51b463b"} Jul 28 14:27:08.336919 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:08.336561 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:08.359506 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:08.359457 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" podStartSLOduration=1.515960695 podStartE2EDuration="6.359437762s" totalImagesPullingTime="4.843477067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:27:08.356711248 +0000 UTC m=+530.021953070" watchObservedRunningTime="2026-07-28 14:27:08.359437762 +0000 UTC m=+530.024679586" Jul 28 14:27:14.044373 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:14.044296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:14.616300 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:14.616269 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:19.344094 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:19.344059 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6h9fn" Jul 28 14:27:23.147459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:23.147418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:28.361505 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:28.361469 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:28.461556 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:28.461522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:28.888697 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:28.885109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:41.372122 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:41.372084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:42.175814 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:42.175779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:49.239703 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:49.239652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:49.362543 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:49.362513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:49.463279 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:49.463249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:49.563435 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:49.563401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:52.847965 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:52.847915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:27:52.976920 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:27:52.976893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:14.062512 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:14.062466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:14.551410 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:14.551376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:18.801151 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:18.801121 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:28:18.804392 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:18.804369 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:28:26.290651 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:26.290617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:26.437016 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:26.436975 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:26.852428 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:26.852394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:27.738047 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:27.738003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:27.844061 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:27.844021 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:28:27.862147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:27.862107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:37.358948 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:37.358905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:37.449184 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:37.449130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:39.610585 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:39.610552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:42.250321 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:42.250238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:42.352981 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:42.352931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:44.482141 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:44.482109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:44.992336 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:44.992304 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:28:49.680079 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:49.680034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:55.520689 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:55.520635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:28:56.193797 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:28:56.193759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:29:12.825829 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:12.825786 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:14.012303 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:14.012269 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:29:19.749028 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.748997 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9"] Jul 28 14:29:19.752343 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.752320 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.755602 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.755571 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 14:29:19.755783 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.755619 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 28 14:29:19.755783 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.755742 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 14:29:19.755783 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.755753 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2p4xs\"" Jul 28 14:29:19.766403 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.766380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9"] Jul 28 14:29:19.810031 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.810007 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:19.848602 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.848577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.848727 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.848612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntddr\" (UniqueName: \"kubernetes.io/projected/167625da-ba48-49ce-9a4d-647f73df47bb-kube-api-access-ntddr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.848727 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.848635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.848727 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.848703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.848881 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.848736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167625da-ba48-49ce-9a4d-647f73df47bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.848881 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.848817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950081 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950173 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950106 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950267 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntddr\" (UniqueName: \"kubernetes.io/projected/167625da-ba48-49ce-9a4d-647f73df47bb-kube-api-access-ntddr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950356 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950356 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167625da-ba48-49ce-9a4d-647f73df47bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950623 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950717 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950670 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.950842 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.950826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.952602 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.952578 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/167625da-ba48-49ce-9a4d-647f73df47bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.952902 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.952886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/167625da-ba48-49ce-9a4d-647f73df47bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:19.959095 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:19.959077 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntddr\" (UniqueName: \"kubernetes.io/projected/167625da-ba48-49ce-9a4d-647f73df47bb-kube-api-access-ntddr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9\" (UID: \"167625da-ba48-49ce-9a4d-647f73df47bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:20.064611 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:20.064584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:20.194250 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:20.194225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9"] Jul 28 14:29:20.196026 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:29:20.195996 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167625da_ba48_49ce_9a4d_647f73df47bb.slice/crio-117ba1675b67737763c16f5240c0b82cad42924736675df52e75c24bf3bf4dc4 WatchSource:0}: Error finding container 117ba1675b67737763c16f5240c0b82cad42924736675df52e75c24bf3bf4dc4: Status 404 returned error can't find the container with id 117ba1675b67737763c16f5240c0b82cad42924736675df52e75c24bf3bf4dc4 Jul 28 14:29:20.197375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:20.197355 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 14:29:21.010771 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:21.010736 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" event={"ID":"167625da-ba48-49ce-9a4d-647f73df47bb","Type":"ContainerStarted","Data":"117ba1675b67737763c16f5240c0b82cad42924736675df52e75c24bf3bf4dc4"} Jul 28 14:29:21.194149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:21.194113 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:29:23.022544 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:23.022511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" event={"ID":"167625da-ba48-49ce-9a4d-647f73df47bb","Type":"ContainerStarted","Data":"daa52a7786c6ad784cec83000149b58c15b46c6d7cc90f59b5030ffd74c5b5b2"} Jul 28 14:29:23.046394 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:23.046341 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" podStartSLOduration=1.8691920770000001 podStartE2EDuration="4.046322295s" totalImagesPullingTime="2.177130218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:29:23.045536217 +0000 UTC m=+664.710778050" watchObservedRunningTime="2026-07-28 14:29:23.046322295 +0000 UTC m=+664.711564119" Jul 28 14:29:30.065344 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:30.065311 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:30.065783 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:30.065375 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:30.077754 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:30.077733 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:31.074830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:31.074802 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9" Jul 28 14:29:43.805095 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:43.805045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:45.799357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:45.799315 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:29:48.729525 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:48.729477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:48.830049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:48.830019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:53.650226 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:53.650187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:53.719570 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:53.719537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:29:58.937751 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:29:58.937717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:30:00.163429 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:00.163402 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq"] Jul 28 14:30:00.167470 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:00.167452 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" Jul 28 14:30:00.170292 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:00.170273 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:30:00.176606 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:00.176590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" Jul 28 14:30:00.183664 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:00.183633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq"] Jul 28 14:30:00.316904 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:00.316880 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq"] Jul 28 14:30:00.319203 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:30:00.319175 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808c9b69_75e0_4232_a5bb_0b7d244aabad.slice/crio-e63b45ff295885d8d7408f762e2109ad5e505aa43ef4e44df26d4189a42e6a46 WatchSource:0}: Error finding container e63b45ff295885d8d7408f762e2109ad5e505aa43ef4e44df26d4189a42e6a46: Status 404 returned error can't find the container with id e63b45ff295885d8d7408f762e2109ad5e505aa43ef4e44df26d4189a42e6a46 Jul 28 14:30:01.220813 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:01.220779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" event={"ID":"808c9b69-75e0-4232-a5bb-0b7d244aabad","Type":"ContainerStarted","Data":"e63b45ff295885d8d7408f762e2109ad5e505aa43ef4e44df26d4189a42e6a46"} Jul 28 14:30:01.408187 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:01.408158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:30:03.233279 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:03.233247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" event={"ID":"808c9b69-75e0-4232-a5bb-0b7d244aabad","Type":"ContainerStarted","Data":"7e92c183c745eb2be6c1b5ea785b54dc5fdbe670f0d45243f6bc60d2cc4e69b3"} Jul 28 14:30:04.239989 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:04.239951 2562 generic.go:356] "Generic (PLEG): container finished" podID="808c9b69-75e0-4232-a5bb-0b7d244aabad" containerID="7e92c183c745eb2be6c1b5ea785b54dc5fdbe670f0d45243f6bc60d2cc4e69b3" exitCode=0 Jul 28 14:30:04.240312 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:04.240016 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" event={"ID":"808c9b69-75e0-4232-a5bb-0b7d244aabad","Type":"ContainerDied","Data":"7e92c183c745eb2be6c1b5ea785b54dc5fdbe670f0d45243f6bc60d2cc4e69b3"} Jul 28 14:30:05.377572 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:05.377548 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" Jul 28 14:30:06.251191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:06.251160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" event={"ID":"808c9b69-75e0-4232-a5bb-0b7d244aabad","Type":"ContainerDied","Data":"e63b45ff295885d8d7408f762e2109ad5e505aa43ef4e44df26d4189a42e6a46"} Jul 28 14:30:06.251191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:06.251182 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq" Jul 28 14:30:06.251377 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:06.251209 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63b45ff295885d8d7408f762e2109ad5e505aa43ef4e44df26d4189a42e6a46" Jul 28 14:30:59.102791 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:59.102757 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:30:59.203517 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:30:59.203489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:31:00.235238 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:31:00.235191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:31:00.354593 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:31:00.354564 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:31:25.087214 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:31:25.087163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:31:34.999176 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:31:34.999142 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:31:45.104417 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:31:45.104319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:31:53.998807 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:31:53.998777 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:32:43.746867 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:32:43.746831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:32:43.977518 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:32:43.977477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:32:44.033787 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:32:44.033693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:33:11.064975 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:11.064861 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:33:12.370518 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:12.370470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:33:18.837938 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:18.837911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:33:18.838768 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:18.838749 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:33:36.909194 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:36.909159 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:33:48.687848 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:48.687807 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:33:56.910262 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:56.910227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:33:57.007196 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:57.007160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:33:57.158337 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:33:57.158308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:03.417974 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:03.417928 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:03.499955 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:03.499913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:03.597158 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:03.597133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:03.693354 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:03.693274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:03.794910 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:03.794879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:09.470502 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:09.470465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:09.570759 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:09.570728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:16.506142 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:16.506104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:34:26.487437 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:26.487396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:34:36.294843 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:36.294801 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:34:44.892504 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:44.892420 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:34:54.085692 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:34:54.085631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:35:05.193533 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:35:05.193499 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:35:16.596667 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:35:16.596625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:35:28.287304 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:35:28.287271 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:35:37.894710 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:35:37.894664 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:35:57.561938 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:35:57.561905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:36:00.792342 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:36:00.792312 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:36:11.685163 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:36:11.685084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:36:36.393899 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:36:36.393858 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:36:47.594648 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:36:47.594613 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:36:58.387505 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:36:58.387462 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:37:04.188416 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:37:04.188366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:37:15.591647 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:37:15.591605 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:37:27.896686 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:37:27.896631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:37:42.191343 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:37:42.191256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:38:18.873891 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:38:18.873865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:38:18.876821 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:38:18.876803 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:39:07.806082 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:39:07.806046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:39:07.889935 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:39:07.889899 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:39:41.796000 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:39:41.795911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:39:51.588733 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:39:51.588692 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:40:06.194387 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:06.194345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:40:15.282350 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:15.282314 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:40:26.790670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:26.790629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:40:34.788876 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:34.788834 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:40:45.183508 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:45.183414 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:40:47.923601 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:47.923566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:40:56.089085 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:40:56.089025 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:41:05.990149 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:05.990108 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:41:12.979002 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:12.978948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:41:15.489242 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:15.489205 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:41:27.484563 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:27.484520 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:41:35.001535 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:35.001498 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:41:45.329665 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:45.329608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:41:53.095556 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:41:53.095515 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:42:02.789472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:02.789436 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:42:12.664668 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:12.664589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:42:15.185242 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:15.185206 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:42:22.151963 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:22.151928 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:42:33.990210 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:33.990172 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:42:55.785226 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:55.785193 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:42:57.276781 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:57.276745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:42:59.887179 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:42:59.887141 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:43:13.087831 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:13.087794 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:43:18.906266 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:18.906243 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:43:18.909277 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:18.909257 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:43:22.305788 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:22.305748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:43:33.987212 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:33.987166 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:43:43.480873 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:43.480784 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:43:52.496926 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:43:52.496885 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:44:02.560520 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:02.560472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:44:02.696931 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:02.696899 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:44:12.083635 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:12.083598 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:44:21.688596 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:21.688555 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:44:34.992074 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:34.992032 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:44:43.994754 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:43.994713 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:44:53.886571 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:44:53.886530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:45:00.135506 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.135467 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t"] Jul 28 14:45:00.136088 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.135993 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808c9b69-75e0-4232-a5bb-0b7d244aabad" containerName="cleanup" Jul 28 14:45:00.136088 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.136015 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="808c9b69-75e0-4232-a5bb-0b7d244aabad" containerName="cleanup" Jul 28 14:45:00.136207 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.136104 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="808c9b69-75e0-4232-a5bb-0b7d244aabad" containerName="cleanup" Jul 28 14:45:00.139121 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.139099 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" Jul 28 14:45:00.141337 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.141313 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:45:00.148379 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.148360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" Jul 28 14:45:00.155202 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.155178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t"] Jul 28 14:45:00.277345 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.277318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t"] Jul 28 14:45:00.280288 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:45:00.280263 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb72740a_174a_426e_9c86_6644b275a578.slice/crio-98cb26dc86531b7978ef8078f1a352e35d8b89d38422c62286f38fd1526bc6d4 WatchSource:0}: Error finding container 98cb26dc86531b7978ef8078f1a352e35d8b89d38422c62286f38fd1526bc6d4: Status 404 returned error can't find the container with id 98cb26dc86531b7978ef8078f1a352e35d8b89d38422c62286f38fd1526bc6d4 Jul 28 14:45:00.992668 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.992637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" event={"ID":"db72740a-174a-426e-9c86-6644b275a578","Type":"ContainerStarted","Data":"27e7d7d2f346a6ab03567ba99affd822ae83b2298c43cde3ca697ea905624749"} Jul 28 14:45:00.992830 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:00.992711 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" event={"ID":"db72740a-174a-426e-9c86-6644b275a578","Type":"ContainerStarted","Data":"98cb26dc86531b7978ef8078f1a352e35d8b89d38422c62286f38fd1526bc6d4"} Jul 28 14:45:01.007821 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:01.007778 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" podStartSLOduration=1.007763735 podStartE2EDuration="1.007763735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:45:01.006659274 +0000 UTC m=+1602.671901085" watchObservedRunningTime="2026-07-28 14:45:01.007763735 +0000 UTC m=+1602.673005547" Jul 28 14:45:01.998735 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:01.998697 2562 generic.go:356] "Generic (PLEG): container finished" podID="db72740a-174a-426e-9c86-6644b275a578" containerID="27e7d7d2f346a6ab03567ba99affd822ae83b2298c43cde3ca697ea905624749" exitCode=0 Jul 28 14:45:01.999059 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:01.998776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" event={"ID":"db72740a-174a-426e-9c86-6644b275a578","Type":"ContainerDied","Data":"27e7d7d2f346a6ab03567ba99affd822ae83b2298c43cde3ca697ea905624749"} Jul 28 14:45:03.135980 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:03.135956 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" Jul 28 14:45:03.284951 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:03.284892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:45:04.011952 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:04.011930 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" Jul 28 14:45:04.012101 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:04.011961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754165-d6b2t" event={"ID":"db72740a-174a-426e-9c86-6644b275a578","Type":"ContainerDied","Data":"98cb26dc86531b7978ef8078f1a352e35d8b89d38422c62286f38fd1526bc6d4"} Jul 28 14:45:04.012101 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:04.012007 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98cb26dc86531b7978ef8078f1a352e35d8b89d38422c62286f38fd1526bc6d4" Jul 28 14:45:12.388980 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:12.388938 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:45:22.984825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:22.984788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:45:40.891212 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:40.891174 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:45:48.899884 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:48.899853 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:45:59.687692 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:45:59.687651 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:46:08.089335 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:08.089297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:46:09.947225 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:09.947187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:21.588443 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:21.588403 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:46:31.294220 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:31.294176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:46:32.104783 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:32.104751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:35.565158 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:35.565122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:35.662112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:35.662076 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:36.134031 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:36.133987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:40.590816 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:40.590724 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:46:42.042302 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.042262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:42.153837 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.153801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:42.338637 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.338602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:42.625206 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.625124 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7"] Jul 28 14:46:42.625541 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.625525 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db72740a-174a-426e-9c86-6644b275a578" containerName="cleanup" Jul 28 14:46:42.625589 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.625543 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="db72740a-174a-426e-9c86-6644b275a578" containerName="cleanup" Jul 28 14:46:42.625623 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.625604 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="db72740a-174a-426e-9c86-6644b275a578" containerName="cleanup" Jul 28 14:46:42.628811 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.628793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.631169 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.631148 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b99a3990-openshift-default-dockercfg-54j5b\"" Jul 28 14:46:42.631292 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.631197 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 14:46:42.649077 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.649041 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:42.650971 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.650950 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7"] Jul 28 14:46:42.727841 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.727817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-envoy\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.727931 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.727844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-data\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.727931 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.727867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-socket\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.727931 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.727893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istiod-ca-cert\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.728083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.727940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-token\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.728083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.727964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-credential-socket\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.728083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.728043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-podinfo\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.728215 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.728084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-certs\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.728215 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.728117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzld\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-kube-api-access-tbzld\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828383 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-credential-socket\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828474 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-podinfo\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828474 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-certs\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828474 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzld\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-kube-api-access-tbzld\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828474 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-envoy\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828704 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-data\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828704 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-socket\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828704 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istiod-ca-cert\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828704 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-token\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828868 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-credential-socket\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828963 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828909 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-certs\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.828963 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.828927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-data\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.829060 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.829007 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-socket\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.829475 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.829455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istiod-ca-cert\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.830872 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.830855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-envoy\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.831002 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.830987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-podinfo\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.835757 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.835731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzld\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-kube-api-access-tbzld\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.835905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.835886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-token\") pod \"e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.939104 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.939053 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:42.965193 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:42.965157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:43.070732 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.070711 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7"] Jul 28 14:46:43.073014 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:46:43.072990 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629fb8ed_6708_4fbc_865e_6d5f6ff91743.slice/crio-a5db006453cc6764177d39ab8588739dc57b12d759ac2b34fa905fb0c8b42a09 WatchSource:0}: Error finding container a5db006453cc6764177d39ab8588739dc57b12d759ac2b34fa905fb0c8b42a09: Status 404 returned error can't find the container with id a5db006453cc6764177d39ab8588739dc57b12d759ac2b34fa905fb0c8b42a09 Jul 28 14:46:43.075147 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.075112 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:46:43.075230 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.075184 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:46:43.075230 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.075212 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:46:43.546595 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.546563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" event={"ID":"629fb8ed-6708-4fbc-865e-6d5f6ff91743","Type":"ContainerStarted","Data":"7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f"} Jul 28 14:46:43.546797 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.546657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" event={"ID":"629fb8ed-6708-4fbc-865e-6d5f6ff91743","Type":"ContainerStarted","Data":"a5db006453cc6764177d39ab8588739dc57b12d759ac2b34fa905fb0c8b42a09"} Jul 28 14:46:43.567222 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.567175 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podStartSLOduration=1.567160538 podStartE2EDuration="1.567160538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:46:43.565914241 +0000 UTC m=+1705.231156063" watchObservedRunningTime="2026-07-28 14:46:43.567160538 +0000 UTC m=+1705.232402435" Jul 28 14:46:43.939215 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:43.939146 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:44.932725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:44.932695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:44.944046 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:44.944023 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:45.013152 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.013123 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:45.080836 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.080793 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:45.220645 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.220582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:45.558125 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.558098 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:45.558957 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.558937 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:46:45.574175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.574146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:45.666522 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:45.666493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:46:51.489166 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:51.489133 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:46:59.645399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:59.645356 2562 generic.go:356] "Generic (PLEG): container finished" podID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerID="7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f" exitCode=0 Jul 28 14:46:59.645838 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:59.645459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" event={"ID":"629fb8ed-6708-4fbc-865e-6d5f6ff91743","Type":"ContainerDied","Data":"7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f"} Jul 28 14:46:59.645901 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:59.645868 2562 scope.go:117] "RemoveContainer" containerID="7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f" Jul 28 14:46:59.646961 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:59.646924 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:46:59.647036 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:59.647009 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:46:59.647085 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:46:59.647041 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:47:00.558534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:00.558497 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:00.653723 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:00.653662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" event={"ID":"629fb8ed-6708-4fbc-865e-6d5f6ff91743","Type":"ContainerStarted","Data":"ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350"} Jul 28 14:47:00.939420 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:00.939337 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:00.940106 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:00.940083 2562 patch_prober.go:28] interesting pod/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 14:47:00.940200 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:00.940129 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 14:47:01.940391 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:01.940360 2562 patch_prober.go:28] interesting pod/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 14:47:01.940755 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:01.940416 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 14:47:02.939422 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:02.939396 2562 patch_prober.go:28] interesting pod/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 14:47:02.939565 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:02.939440 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 14:47:03.939946 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:03.939911 2562 patch_prober.go:28] interesting pod/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 28 14:47:03.940321 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:03.939970 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 28 14:47:04.943347 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:04.943317 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:04.943716 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:04.943653 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:04.944288 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:04.944267 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:10.802995 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.802957 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp"] Jul 28 14:47:10.807041 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.807006 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:10.810352 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.810327 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:47:10.810539 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.810362 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 14:47:10.816428 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.816399 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp"] Jul 28 14:47:10.850562 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.850539 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e99010-968f-44e4-81b7-8678db20c0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp\" (UID: \"83e99010-968f-44e4-81b7-8678db20c0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:10.951566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.951542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e99010-968f-44e4-81b7-8678db20c0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp\" (UID: \"83e99010-968f-44e4-81b7-8678db20c0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:10.952146 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:10.952127 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e99010-968f-44e4-81b7-8678db20c0a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp\" (UID: \"83e99010-968f-44e4-81b7-8678db20c0a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:11.118012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:11.117961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:11.248019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:11.247992 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp"] Jul 28 14:47:11.249112 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:47:11.249085 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e99010_968f_44e4_81b7_8678db20c0a3.slice/crio-7d8c1ec10374b993f044aa84bc2c8092b04a3d549d40b009ecf1ef98f5d55e9a WatchSource:0}: Error finding container 7d8c1ec10374b993f044aa84bc2c8092b04a3d549d40b009ecf1ef98f5d55e9a: Status 404 returned error can't find the container with id 7d8c1ec10374b993f044aa84bc2c8092b04a3d549d40b009ecf1ef98f5d55e9a Jul 28 14:47:11.713325 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:11.713262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" event={"ID":"83e99010-968f-44e4-81b7-8678db20c0a3","Type":"ContainerStarted","Data":"623fcd04282234211a8a016871c65f8439def01b20f9d7d2dd4ec946dcfbdf7f"} Jul 28 14:47:11.713325 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:11.713314 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" event={"ID":"83e99010-968f-44e4-81b7-8678db20c0a3","Type":"ContainerStarted","Data":"7d8c1ec10374b993f044aa84bc2c8092b04a3d549d40b009ecf1ef98f5d55e9a"} Jul 28 14:47:11.731249 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:11.731203 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" podStartSLOduration=1.731189338 podStartE2EDuration="1.731189338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:47:11.728601877 +0000 UTC m=+1733.393843697" watchObservedRunningTime="2026-07-28 14:47:11.731189338 +0000 UTC m=+1733.396431159" Jul 28 14:47:12.720357 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:12.720325 2562 generic.go:356] "Generic (PLEG): container finished" podID="83e99010-968f-44e4-81b7-8678db20c0a3" containerID="623fcd04282234211a8a016871c65f8439def01b20f9d7d2dd4ec946dcfbdf7f" exitCode=0 Jul 28 14:47:12.720730 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:12.720374 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" event={"ID":"83e99010-968f-44e4-81b7-8678db20c0a3","Type":"ContainerDied","Data":"623fcd04282234211a8a016871c65f8439def01b20f9d7d2dd4ec946dcfbdf7f"} Jul 28 14:47:13.853944 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:13.853914 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:13.974261 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:13.974237 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e99010-968f-44e4-81b7-8678db20c0a3-maas-api-service-ca\") pod \"83e99010-968f-44e4-81b7-8678db20c0a3\" (UID: \"83e99010-968f-44e4-81b7-8678db20c0a3\") " Jul 28 14:47:13.974541 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:13.974518 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e99010-968f-44e4-81b7-8678db20c0a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83e99010-968f-44e4-81b7-8678db20c0a3" (UID: "83e99010-968f-44e4-81b7-8678db20c0a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:47:14.075686 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:14.075653 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e99010-968f-44e4-81b7-8678db20c0a3-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:14.733185 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:14.733157 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" Jul 28 14:47:14.733318 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:14.733184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp" event={"ID":"83e99010-968f-44e4-81b7-8678db20c0a3","Type":"ContainerDied","Data":"7d8c1ec10374b993f044aa84bc2c8092b04a3d549d40b009ecf1ef98f5d55e9a"} Jul 28 14:47:14.733318 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:14.733233 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d8c1ec10374b993f044aa84bc2c8092b04a3d549d40b009ecf1ef98f5d55e9a" Jul 28 14:47:15.586947 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.586911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:15.652775 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.652739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:15.720153 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.720124 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:15.757121 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.757088 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp"] Jul 28 14:47:15.759251 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.759229 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b99a3990-bwzjp"] Jul 28 14:47:15.789026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.788994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:15.917759 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.917691 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7"] Jul 28 14:47:15.917924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.917900 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" containerID="cri-o://ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350" gracePeriod=30 Jul 28 14:47:15.940048 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:15.940022 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:16.184809 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:16.184727 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:16.451916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:16.451850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:16.543912 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:16.543888 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:16.722318 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:16.722251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:16.888785 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:16.888759 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e99010-968f-44e4-81b7-8678db20c0a3" path="/var/lib/kubelet/pods/83e99010-968f-44e4-81b7-8678db20c0a3/volumes" Jul 28 14:47:17.085194 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:17.085170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:17.607885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:17.607855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:19.388369 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:19.388340 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:47:19.946349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:19.946307 2562 patch_prober.go:28] interesting pod/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 28 14:47:19.946530 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:19.946373 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 14:47:20.092738 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:20.092705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:20.288521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:20.288437 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:20.491112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:20.491085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:20.851911 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:20.851882 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:20.940541 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:20.940516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:21.169175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.169154 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:21.231916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.231887 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-certs\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.231925 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-socket\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.231954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-envoy\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232003 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbzld\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-kube-api-access-tbzld\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232049 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232043 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-credential-socket\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232085 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-data\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232131 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-podinfo\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232154 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:47:21.232258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232164 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-token\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232192 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:47:21.232258 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232244 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istiod-ca-cert\") pod \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\" (UID: \"629fb8ed-6708-4fbc-865e-6d5f6ff91743\") " Jul 28 14:47:21.232846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232336 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:47:21.232846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232569 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.232846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232589 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.232846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232604 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.232846 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232658 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:47:21.233098 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.232876 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-data" (OuterVolumeSpecName: "istio-data") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:47:21.234244 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.234213 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 14:47:21.234386 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.234259 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-token" (OuterVolumeSpecName: "istio-token") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:47:21.234498 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.234481 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-kube-api-access-tbzld" (OuterVolumeSpecName: "kube-api-access-tbzld") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "kube-api-access-tbzld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:47:21.234556 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.234545 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "629fb8ed-6708-4fbc-865e-6d5f6ff91743" (UID: "629fb8ed-6708-4fbc-865e-6d5f6ff91743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:47:21.333667 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.333646 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.333667 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.333665 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.333803 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.333689 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbzld\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-kube-api-access-tbzld\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.333803 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.333698 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.333803 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.333707 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.333803 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.333714 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629fb8ed-6708-4fbc-865e-6d5f6ff91743-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:21.771166 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.771139 2562 generic.go:356] "Generic (PLEG): container finished" podID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerID="ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350" exitCode=0 Jul 28 14:47:21.771523 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.771183 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" Jul 28 14:47:21.771523 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.771228 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" event={"ID":"629fb8ed-6708-4fbc-865e-6d5f6ff91743","Type":"ContainerDied","Data":"ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350"} Jul 28 14:47:21.771523 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.771279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7" event={"ID":"629fb8ed-6708-4fbc-865e-6d5f6ff91743","Type":"ContainerDied","Data":"a5db006453cc6764177d39ab8588739dc57b12d759ac2b34fa905fb0c8b42a09"} Jul 28 14:47:21.771523 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.771301 2562 scope.go:117] "RemoveContainer" containerID="ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350" Jul 28 14:47:21.780881 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.780864 2562 scope.go:117] "RemoveContainer" containerID="7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f" Jul 28 14:47:21.789114 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.789100 2562 scope.go:117] "RemoveContainer" containerID="ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350" Jul 28 14:47:21.789357 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:47:21.789338 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350\": container with ID starting with ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350 not found: ID does not exist" containerID="ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350" Jul 28 14:47:21.789423 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.789364 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350"} err="failed to get container status \"ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350\": rpc error: code = NotFound desc = could not find container \"ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350\": container with ID starting with ff38e6a8b94b0022113529ded34b877e54f50ca95425fd15ce3d3238f13ad350 not found: ID does not exist" Jul 28 14:47:21.789423 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.789385 2562 scope.go:117] "RemoveContainer" containerID="7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f" Jul 28 14:47:21.789634 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:47:21.789619 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f\": container with ID starting with 7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f not found: ID does not exist" containerID="7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f" Jul 28 14:47:21.789670 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.789641 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f"} err="failed to get container status \"7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f\": rpc error: code = NotFound desc = could not find container \"7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f\": container with ID starting with 7411e19e87d8d61182ab2add8c6b1453882a28ffe94277bbb1a6835ac4f51c9f not found: ID does not exist" Jul 28 14:47:21.793760 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.793741 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7"] Jul 28 14:47:21.799005 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:21.798987 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b99a3990-openshift-default-7d9cdffd88-8scv7"] Jul 28 14:47:22.888713 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:22.888659 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" path="/var/lib/kubelet/pods/629fb8ed-6708-4fbc-865e-6d5f6ff91743/volumes" Jul 28 14:47:28.788622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:28.788583 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:47:45.212597 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212558 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4"] Jul 28 14:47:45.213012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212912 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e99010-968f-44e4-81b7-8678db20c0a3" containerName="revoke-keys" Jul 28 14:47:45.213012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212926 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e99010-968f-44e4-81b7-8678db20c0a3" containerName="revoke-keys" Jul 28 14:47:45.213012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212947 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" Jul 28 14:47:45.213012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212953 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" Jul 28 14:47:45.213012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212960 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" Jul 28 14:47:45.213012 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.212965 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" Jul 28 14:47:45.213247 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.213019 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" Jul 28 14:47:45.213247 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.213028 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e99010-968f-44e4-81b7-8678db20c0a3" containerName="revoke-keys" Jul 28 14:47:45.217823 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.217800 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:45.221051 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.221023 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:47:45.221204 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.221070 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 14:47:45.223612 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.223587 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4"] Jul 28 14:47:45.310452 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.310422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b5f925-8162-449e-974f-05dd56b3542a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4\" (UID: \"04b5f925-8162-449e-974f-05dd56b3542a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:45.411780 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.411749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b5f925-8162-449e-974f-05dd56b3542a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4\" (UID: \"04b5f925-8162-449e-974f-05dd56b3542a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:45.412356 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.412334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b5f925-8162-449e-974f-05dd56b3542a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4\" (UID: \"04b5f925-8162-449e-974f-05dd56b3542a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:45.531806 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.531742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:45.663511 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.663477 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4"] Jul 28 14:47:45.667255 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:47:45.667232 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b5f925_8162_449e_974f_05dd56b3542a.slice/crio-43e539355d20766526e82a2f04ffad9cd1959f83d836b60d749aa861c4fc5c69 WatchSource:0}: Error finding container 43e539355d20766526e82a2f04ffad9cd1959f83d836b60d749aa861c4fc5c69: Status 404 returned error can't find the container with id 43e539355d20766526e82a2f04ffad9cd1959f83d836b60d749aa861c4fc5c69 Jul 28 14:47:45.914215 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:45.914191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" event={"ID":"04b5f925-8162-449e-974f-05dd56b3542a","Type":"ContainerStarted","Data":"43e539355d20766526e82a2f04ffad9cd1959f83d836b60d749aa861c4fc5c69"} Jul 28 14:47:46.921300 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:46.921267 2562 generic.go:356] "Generic (PLEG): container finished" podID="04b5f925-8162-449e-974f-05dd56b3542a" containerID="2068683d07d1fb5c0dfb0750be76cdb8b23a79781d49bc01a417aff5fc1c23d6" exitCode=0 Jul 28 14:47:46.921724 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:46.921323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" event={"ID":"04b5f925-8162-449e-974f-05dd56b3542a","Type":"ContainerDied","Data":"2068683d07d1fb5c0dfb0750be76cdb8b23a79781d49bc01a417aff5fc1c23d6"} Jul 28 14:47:48.062427 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.062407 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:48.234329 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.234261 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b5f925-8162-449e-974f-05dd56b3542a-maas-api-service-ca\") pod \"04b5f925-8162-449e-974f-05dd56b3542a\" (UID: \"04b5f925-8162-449e-974f-05dd56b3542a\") " Jul 28 14:47:48.234578 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.234557 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b5f925-8162-449e-974f-05dd56b3542a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04b5f925-8162-449e-974f-05dd56b3542a" (UID: "04b5f925-8162-449e-974f-05dd56b3542a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:47:48.334975 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.334942 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04b5f925-8162-449e-974f-05dd56b3542a-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:47:48.933563 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.933540 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" Jul 28 14:47:48.933708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.933585 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4" event={"ID":"04b5f925-8162-449e-974f-05dd56b3542a","Type":"ContainerDied","Data":"43e539355d20766526e82a2f04ffad9cd1959f83d836b60d749aa861c4fc5c69"} Jul 28 14:47:48.933708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:48.933632 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e539355d20766526e82a2f04ffad9cd1959f83d836b60d749aa861c4fc5c69" Jul 28 14:47:51.530316 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.530272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:51.601133 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.601103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:51.713933 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.713893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:51.765607 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.765580 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4"] Jul 28 14:47:51.771322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.771303 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35b9d8ff-dszr4"] Jul 28 14:47:51.798389 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.798369 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:51.942996 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:51.942947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:52.223591 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.223509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:52.525120 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.525054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:52.827199 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.827171 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb"] Jul 28 14:47:52.827619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.827602 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b5f925-8162-449e-974f-05dd56b3542a" containerName="revoke-keys" Jul 28 14:47:52.827619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.827620 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b5f925-8162-449e-974f-05dd56b3542a" containerName="revoke-keys" Jul 28 14:47:52.827749 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.827725 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="629fb8ed-6708-4fbc-865e-6d5f6ff91743" containerName="istio-proxy" Jul 28 14:47:52.827749 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.827733 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b5f925-8162-449e-974f-05dd56b3542a" containerName="revoke-keys" Jul 28 14:47:52.830723 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.830710 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.832974 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.832956 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 14:47:52.833042 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.833027 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1bf67f1c-openshift-default-dockercfg-jx9qz\"" Jul 28 14:47:52.845184 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.845155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb"] Jul 28 14:47:52.865608 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.865577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:52.888641 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.888617 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b5f925-8162-449e-974f-05dd56b3542a" path="/var/lib/kubelet/pods/04b5f925-8162-449e-974f-05dd56b3542a/volumes" Jul 28 14:47:52.968267 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-envoy\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968378 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2625067f-207f-4051-b97a-aeb12f84f5fd-istiod-ca-cert\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968378 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lhl\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-kube-api-access-f4lhl\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968378 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-data\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968480 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968405 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-podinfo\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968480 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-token\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968480 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-certs\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968477 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-socket\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:52.968567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:52.968498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-credential-socket\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069317 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-podinfo\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069408 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-token\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069408 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-certs\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-socket\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069534 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-credential-socket\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069647 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-envoy\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069737 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2625067f-207f-4051-b97a-aeb12f84f5fd-istiod-ca-cert\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069804 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lhl\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-kube-api-access-f4lhl\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069804 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-data\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069894 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-credential-socket\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.069894 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.069865 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-socket\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.070095 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.070072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-certs\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.070156 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.070110 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-data\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.070323 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.070306 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2625067f-207f-4051-b97a-aeb12f84f5fd-istiod-ca-cert\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.071825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.071804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-envoy\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.071924 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.071862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-podinfo\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.079714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.079658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lhl\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-kube-api-access-f4lhl\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.082528 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.082503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-token\") pod \"e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.140501 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.140480 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:53.219613 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.219573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:53.274866 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.274844 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb"] Jul 28 14:47:53.276877 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:47:53.276856 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2625067f_207f_4051_b97a_aeb12f84f5fd.slice/crio-6020e20bae4aedea1fd8c0c043a5b342d125ee82218314ff8781536b6f349f7f WatchSource:0}: Error finding container 6020e20bae4aedea1fd8c0c043a5b342d125ee82218314ff8781536b6f349f7f: Status 404 returned error can't find the container with id 6020e20bae4aedea1fd8c0c043a5b342d125ee82218314ff8781536b6f349f7f Jul 28 14:47:53.278847 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.278820 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:47:53.278918 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.278876 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:47:53.278918 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.278903 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:47:53.962378 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.962344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" event={"ID":"2625067f-207f-4051-b97a-aeb12f84f5fd","Type":"ContainerStarted","Data":"1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466"} Jul 28 14:47:53.962769 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.962425 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" event={"ID":"2625067f-207f-4051-b97a-aeb12f84f5fd","Type":"ContainerStarted","Data":"6020e20bae4aedea1fd8c0c043a5b342d125ee82218314ff8781536b6f349f7f"} Jul 28 14:47:53.985880 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:53.985835 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podStartSLOduration=1.985820208 podStartE2EDuration="1.985820208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:47:53.982957724 +0000 UTC m=+1775.648199545" watchObservedRunningTime="2026-07-28 14:47:53.985820208 +0000 UTC m=+1775.651062030" Jul 28 14:47:54.140973 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:54.140949 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:54.145399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:54.145371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:54.967731 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:54.967700 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:54.968437 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:54.968422 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:47:55.360866 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:55.360834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:55.455099 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:55.455069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:55.624953 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:55.624864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:55.800432 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:55.800401 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:47:55.969138 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:55.969060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:56.308238 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:56.308211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:47:56.399945 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:47:56.399908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:05.091813 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:05.091782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:48:09.057647 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.057609 2562 generic.go:356] "Generic (PLEG): container finished" podID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerID="1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466" exitCode=0 Jul 28 14:48:09.058145 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.057697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" event={"ID":"2625067f-207f-4051-b97a-aeb12f84f5fd","Type":"ContainerDied","Data":"1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466"} Jul 28 14:48:09.058145 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.058104 2562 scope.go:117] "RemoveContainer" containerID="1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466" Jul 28 14:48:09.059327 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.059284 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:48:09.059412 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.059391 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:48:09.059448 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.059439 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:48:09.968048 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:09.968016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:10.064650 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:10.064612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" event={"ID":"2625067f-207f-4051-b97a-aeb12f84f5fd","Type":"ContainerStarted","Data":"7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0"} Jul 28 14:48:10.140689 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:10.140653 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:10.141639 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:10.141611 2562 patch_prober.go:28] interesting pod/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 28 14:48:10.141778 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:10.141660 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 28 14:48:11.140820 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:11.140740 2562 patch_prober.go:28] interesting pod/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 28 14:48:11.140820 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:11.140794 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 28 14:48:12.140895 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:12.140864 2562 patch_prober.go:28] interesting pod/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 28 14:48:12.141240 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:12.140914 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 28 14:48:13.141008 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:13.140970 2562 patch_prober.go:28] interesting pod/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 28 14:48:13.141399 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:13.141034 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 28 14:48:14.144544 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:14.144508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:14.144942 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:14.144776 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:14.145334 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:14.145317 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:18.940450 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:18.940425 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:48:18.944353 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:18.944334 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:48:21.238431 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.238385 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8"] Jul 28 14:48:21.241958 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.241933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:21.244115 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.244093 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:48:21.244211 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.244187 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 14:48:21.248803 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.248777 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8"] Jul 28 14:48:21.291914 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.291894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42accd-b2a8-4df0-b918-c7277807f514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8\" (UID: \"2c42accd-b2a8-4df0-b918-c7277807f514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:21.392391 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.392359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42accd-b2a8-4df0-b918-c7277807f514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8\" (UID: \"2c42accd-b2a8-4df0-b918-c7277807f514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:21.392956 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.392935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42accd-b2a8-4df0-b918-c7277807f514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8\" (UID: \"2c42accd-b2a8-4df0-b918-c7277807f514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:21.554659 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.554627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:21.682182 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:21.682158 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8"] Jul 28 14:48:21.684011 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:48:21.683985 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c42accd_b2a8_4df0_b918_c7277807f514.slice/crio-9306c9c2329b797264514831424a6534409385d6264b1399bc9cdbbf502dd437 WatchSource:0}: Error finding container 9306c9c2329b797264514831424a6534409385d6264b1399bc9cdbbf502dd437: Status 404 returned error can't find the container with id 9306c9c2329b797264514831424a6534409385d6264b1399bc9cdbbf502dd437 Jul 28 14:48:22.134313 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:22.134287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" event={"ID":"2c42accd-b2a8-4df0-b918-c7277807f514","Type":"ContainerStarted","Data":"08bfbd1a0aa8d7537fa8b8eeb961ef087a265e42f95e05fd75e28903e29dbf96"} Jul 28 14:48:22.134455 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:22.134375 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" event={"ID":"2c42accd-b2a8-4df0-b918-c7277807f514","Type":"ContainerStarted","Data":"9306c9c2329b797264514831424a6534409385d6264b1399bc9cdbbf502dd437"} Jul 28 14:48:22.149821 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:22.149779 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" podStartSLOduration=1.149765711 podStartE2EDuration="1.149765711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:48:22.148513881 +0000 UTC m=+1803.813755704" watchObservedRunningTime="2026-07-28 14:48:22.149765711 +0000 UTC m=+1803.815007532" Jul 28 14:48:23.141279 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:23.141246 2562 generic.go:356] "Generic (PLEG): container finished" podID="2c42accd-b2a8-4df0-b918-c7277807f514" containerID="08bfbd1a0aa8d7537fa8b8eeb961ef087a265e42f95e05fd75e28903e29dbf96" exitCode=0 Jul 28 14:48:23.141708 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:23.141343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" event={"ID":"2c42accd-b2a8-4df0-b918-c7277807f514","Type":"ContainerDied","Data":"08bfbd1a0aa8d7537fa8b8eeb961ef087a265e42f95e05fd75e28903e29dbf96"} Jul 28 14:48:24.275897 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:24.275874 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:24.316764 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:24.316739 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42accd-b2a8-4df0-b918-c7277807f514-maas-api-service-ca\") pod \"2c42accd-b2a8-4df0-b918-c7277807f514\" (UID: \"2c42accd-b2a8-4df0-b918-c7277807f514\") " Jul 28 14:48:24.317038 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:24.317018 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c42accd-b2a8-4df0-b918-c7277807f514-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c42accd-b2a8-4df0-b918-c7277807f514" (UID: "2c42accd-b2a8-4df0-b918-c7277807f514"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:48:24.418267 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:24.418202 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c42accd-b2a8-4df0-b918-c7277807f514-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:25.154197 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:25.154122 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" event={"ID":"2c42accd-b2a8-4df0-b918-c7277807f514","Type":"ContainerDied","Data":"9306c9c2329b797264514831424a6534409385d6264b1399bc9cdbbf502dd437"} Jul 28 14:48:25.154197 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:25.154176 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9306c9c2329b797264514831424a6534409385d6264b1399bc9cdbbf502dd437" Jul 28 14:48:25.154197 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:25.154174 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8" Jul 28 14:48:25.952540 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:25.952500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:26.093104 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.093068 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:26.219171 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.219092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:26.293048 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.293021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:26.434428 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.434397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:26.501903 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.501833 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8"] Jul 28 14:48:26.507944 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.507922 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1bf67f1c-kcdf8"] Jul 28 14:48:26.545549 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.545523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:26.888189 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:26.888161 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c42accd-b2a8-4df0-b918-c7277807f514" path="/var/lib/kubelet/pods/2c42accd-b2a8-4df0-b918-c7277807f514/volumes" Jul 28 14:48:27.713629 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:27.713594 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb"] Jul 28 14:48:27.714108 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:27.713854 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" containerID="cri-o://7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0" gracePeriod=30 Jul 28 14:48:27.736337 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:27.736306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:27.985229 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:27.985158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:28.222834 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:28.222798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:28.540714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:28.540655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:28.906535 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:28.906509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:29.146694 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:29.146641 2562 patch_prober.go:28] interesting pod/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 14:48:29.146837 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:29.146718 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 14:48:29.892280 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:29.892242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:48:31.291255 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:31.291227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:31.394741 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:31.394698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:31.474582 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:31.474552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:31.970589 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:31.970556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:32.077547 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:32.077515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:48:32.964878 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:32.964859 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:33.090108 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090082 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2625067f-207f-4051-b97a-aeb12f84f5fd-istiod-ca-cert\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090125 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-certs\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-envoy\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-credential-socket\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090204 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-podinfo\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090248 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090230 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-socket\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090252 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-data\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090280 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-token\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090307 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4lhl\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-kube-api-access-f4lhl\") pod \"2625067f-207f-4051-b97a-aeb12f84f5fd\" (UID: \"2625067f-207f-4051-b97a-aeb12f84f5fd\") " Jul 28 14:48:33.090510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090366 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:48:33.090713 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090507 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2625067f-207f-4051-b97a-aeb12f84f5fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:48:33.090713 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090553 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.090713 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090586 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:48:33.090901 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.090878 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:48:33.091346 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.091317 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:48:33.092467 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.092432 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 14:48:33.092616 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.092549 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-kube-api-access-f4lhl" (OuterVolumeSpecName: "kube-api-access-f4lhl") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "kube-api-access-f4lhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:48:33.092738 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.092720 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:48:33.092795 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.092742 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "2625067f-207f-4051-b97a-aeb12f84f5fd" (UID: "2625067f-207f-4051-b97a-aeb12f84f5fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:48:33.191784 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191756 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191784 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191776 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191784 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191785 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191794 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191802 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191810 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191818 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lhl\" (UniqueName: \"kubernetes.io/projected/2625067f-207f-4051-b97a-aeb12f84f5fd-kube-api-access-f4lhl\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.191991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.191827 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2625067f-207f-4051-b97a-aeb12f84f5fd-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:33.196714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.196669 2562 generic.go:356] "Generic (PLEG): container finished" podID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerID="7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0" exitCode=0 Jul 28 14:48:33.196825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.196742 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" Jul 28 14:48:33.196825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.196774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" event={"ID":"2625067f-207f-4051-b97a-aeb12f84f5fd","Type":"ContainerDied","Data":"7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0"} Jul 28 14:48:33.196903 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.196828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb" event={"ID":"2625067f-207f-4051-b97a-aeb12f84f5fd","Type":"ContainerDied","Data":"6020e20bae4aedea1fd8c0c043a5b342d125ee82218314ff8781536b6f349f7f"} Jul 28 14:48:33.196903 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.196859 2562 scope.go:117] "RemoveContainer" containerID="7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0" Jul 28 14:48:33.212028 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.212006 2562 scope.go:117] "RemoveContainer" containerID="1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466" Jul 28 14:48:33.219807 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.219784 2562 scope.go:117] "RemoveContainer" containerID="7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0" Jul 28 14:48:33.220119 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:48:33.220094 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0\": container with ID starting with 7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0 not found: ID does not exist" containerID="7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0" Jul 28 14:48:33.220196 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.220159 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0"} err="failed to get container status \"7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0\": rpc error: code = NotFound desc = could not find container \"7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0\": container with ID starting with 7717126a2e91da351adc31014a8627b5db80f8d6a52f878533e73f43dbc3aeb0 not found: ID does not exist" Jul 28 14:48:33.220259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.220197 2562 scope.go:117] "RemoveContainer" containerID="1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466" Jul 28 14:48:33.220669 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:48:33.220606 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466\": container with ID starting with 1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466 not found: ID does not exist" containerID="1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466" Jul 28 14:48:33.220669 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.220696 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466"} err="failed to get container status \"1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466\": rpc error: code = NotFound desc = could not find container \"1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466\": container with ID starting with 1b8d3aab510f2a3e914d4bd899e630977bcde3ba3b5752c3de49b34c619b4466 not found: ID does not exist" Jul 28 14:48:33.222548 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.222528 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb"] Jul 28 14:48:33.226753 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:33.226734 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1bf67f1c-openshift-default-74598f8744-hxhbb"] Jul 28 14:48:34.888919 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:34.888881 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" path="/var/lib/kubelet/pods/2625067f-207f-4051-b97a-aeb12f84f5fd/volumes" Jul 28 14:48:40.097239 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:40.097197 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:48:56.239375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239334 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs"] Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239714 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c42accd-b2a8-4df0-b918-c7277807f514" containerName="revoke-keys" Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239729 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c42accd-b2a8-4df0-b918-c7277807f514" containerName="revoke-keys" Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239740 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239745 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239833 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239842 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" Jul 28 14:48:56.239885 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.239848 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c42accd-b2a8-4df0-b918-c7277807f514" containerName="revoke-keys" Jul 28 14:48:56.242902 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.242883 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:56.245476 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.245449 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 14:48:56.245554 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.245534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:48:56.251566 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.251542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs"] Jul 28 14:48:56.361348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.361320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377bcf9-a136-43e9-8397-1a88a86439db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs\" (UID: \"4377bcf9-a136-43e9-8397-1a88a86439db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:56.461817 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.461784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377bcf9-a136-43e9-8397-1a88a86439db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs\" (UID: \"4377bcf9-a136-43e9-8397-1a88a86439db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:56.462382 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.462361 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377bcf9-a136-43e9-8397-1a88a86439db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs\" (UID: \"4377bcf9-a136-43e9-8397-1a88a86439db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:56.557017 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.556986 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:56.691014 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:56.690985 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs"] Jul 28 14:48:56.692292 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:48:56.692269 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4377bcf9_a136_43e9_8397_1a88a86439db.slice/crio-d412704f2913705020650fd64ad6862120d112fc527d94fdecd0980fb0f92b7d WatchSource:0}: Error finding container d412704f2913705020650fd64ad6862120d112fc527d94fdecd0980fb0f92b7d: Status 404 returned error can't find the container with id d412704f2913705020650fd64ad6862120d112fc527d94fdecd0980fb0f92b7d Jul 28 14:48:57.329082 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:57.329055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" event={"ID":"4377bcf9-a136-43e9-8397-1a88a86439db","Type":"ContainerStarted","Data":"aba712be7f10355430077c511b7eba8538bc662084e8276804c9796937a72a10"} Jul 28 14:48:57.329380 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:57.329114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" event={"ID":"4377bcf9-a136-43e9-8397-1a88a86439db","Type":"ContainerStarted","Data":"d412704f2913705020650fd64ad6862120d112fc527d94fdecd0980fb0f92b7d"} Jul 28 14:48:58.336025 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:58.335987 2562 generic.go:356] "Generic (PLEG): container finished" podID="4377bcf9-a136-43e9-8397-1a88a86439db" containerID="aba712be7f10355430077c511b7eba8538bc662084e8276804c9796937a72a10" exitCode=0 Jul 28 14:48:58.336443 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:58.336055 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" event={"ID":"4377bcf9-a136-43e9-8397-1a88a86439db","Type":"ContainerDied","Data":"aba712be7f10355430077c511b7eba8538bc662084e8276804c9796937a72a10"} Jul 28 14:48:58.466020 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:58.465998 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:58.578046 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:58.578025 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377bcf9-a136-43e9-8397-1a88a86439db-maas-api-service-ca\") pod \"4377bcf9-a136-43e9-8397-1a88a86439db\" (UID: \"4377bcf9-a136-43e9-8397-1a88a86439db\") " Jul 28 14:48:58.578372 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:58.578353 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4377bcf9-a136-43e9-8397-1a88a86439db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4377bcf9-a136-43e9-8397-1a88a86439db" (UID: "4377bcf9-a136-43e9-8397-1a88a86439db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:48:58.679572 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:58.679517 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377bcf9-a136-43e9-8397-1a88a86439db-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:48:59.342497 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:59.342473 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" Jul 28 14:48:59.342497 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:59.342480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs" event={"ID":"4377bcf9-a136-43e9-8397-1a88a86439db","Type":"ContainerDied","Data":"d412704f2913705020650fd64ad6862120d112fc527d94fdecd0980fb0f92b7d"} Jul 28 14:48:59.342987 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:48:59.342526 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d412704f2913705020650fd64ad6862120d112fc527d94fdecd0980fb0f92b7d" Jul 28 14:49:02.258771 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.258735 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:02.356697 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.356658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:02.397322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.397295 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs"] Jul 28 14:49:02.401602 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.401581 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e9a43c4a-wtmhs"] Jul 28 14:49:02.450097 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.450071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:02.556849 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.556797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:02.824991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.824918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:02.888348 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:02.888320 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4377bcf9-a136-43e9-8397-1a88a86439db" path="/var/lib/kubelet/pods/4377bcf9-a136-43e9-8397-1a88a86439db/volumes" Jul 28 14:49:03.364201 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:03.364151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:03.453870 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:03.453846 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:03.578724 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:03.578697 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:49:06.302340 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:06.302307 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:49:06.710757 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:49:06.710653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:08.310126 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:08.310078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:14.702745 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:14.702644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:16.486312 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:16.486265 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:30.914785 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:30.914732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:41.653745 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:41.653655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:48.736123 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:48.736089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:48.838507 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:48.838470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:51:48.938555 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:51:48.938521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:05.012423 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:05.012376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:10.863021 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:10.862966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:10.970123 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:10.970092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:11.099459 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:11.099428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:21.654026 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:21.653970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:22.537188 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:22.537153 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:28.971588 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:28.971551 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:29.080918 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:29.080894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:29.171902 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:29.171877 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:29.294720 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:29.294693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:40.852878 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:40.852796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:40.947734 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:40.947703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:52:41.050786 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:52:41.050756 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:53:10.382595 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:53:10.382557 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:53:18.982424 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:53:18.982392 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:53:18.988288 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:53:18.988269 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:53:19.025464 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:53:19.025440 2562 scope.go:117] "RemoveContainer" containerID="623fcd04282234211a8a016871c65f8439def01b20f9d7d2dd4ec946dcfbdf7f" Jul 28 14:53:26.943284 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:53:26.943245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:54:19.047152 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:54:19.047071 2562 scope.go:117] "RemoveContainer" containerID="2068683d07d1fb5c0dfb0750be76cdb8b23a79781d49bc01a417aff5fc1c23d6" Jul 28 14:55:19.072434 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:55:19.072390 2562 scope.go:117] "RemoveContainer" containerID="08bfbd1a0aa8d7537fa8b8eeb961ef087a265e42f95e05fd75e28903e29dbf96" Jul 28 14:55:19.081375 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:55:19.081346 2562 scope.go:117] "RemoveContainer" containerID="aba712be7f10355430077c511b7eba8538bc662084e8276804c9796937a72a10" Jul 28 14:56:17.476521 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:56:17.476481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:56:23.643229 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:56:23.643181 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:48.137843 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:48.137808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:54.592306 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:54.592271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:54.694732 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:54.694699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:54.858158 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:54.858085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:55.172604 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.172525 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z"] Jul 28 14:57:55.173112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.173080 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" Jul 28 14:57:55.173112 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.173103 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2625067f-207f-4051-b97a-aeb12f84f5fd" containerName="istio-proxy" Jul 28 14:57:55.173293 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.173140 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4377bcf9-a136-43e9-8397-1a88a86439db" containerName="revoke-keys" Jul 28 14:57:55.173293 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.173148 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4377bcf9-a136-43e9-8397-1a88a86439db" containerName="revoke-keys" Jul 28 14:57:55.173293 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.173239 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4377bcf9-a136-43e9-8397-1a88a86439db" containerName="revoke-keys" Jul 28 14:57:55.176282 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.176261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.178831 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.178810 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 14:57:55.178917 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.178828 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8b4faed4-openshift-default-dockercfg-7rvk9\"" Jul 28 14:57:55.198212 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.198191 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z"] Jul 28 14:57:55.201071 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.201046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:55.369439 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4ng\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-kube-api-access-lh4ng\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369448 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-data\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-istio-token\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369567 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836a83f6-4979-43ea-b001-72756d2de921-istio-podinfo\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369718 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-socket\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369718 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-certs\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369718 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-envoy\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369718 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836a83f6-4979-43ea-b001-72756d2de921-istiod-ca-cert\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.369869 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.369761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-credential-socket\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.400169 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.400142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:55.470656 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.470592 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-credential-socket\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.470882 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.470856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4ng\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-kube-api-access-lh4ng\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.470903 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-data\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.470943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-istio-token\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471006 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.470996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836a83f6-4979-43ea-b001-72756d2de921-istio-podinfo\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471171 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-credential-socket\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471228 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-socket\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-certs\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471322 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-data\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471506 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-envoy\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471506 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836a83f6-4979-43ea-b001-72756d2de921-istiod-ca-cert\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-socket\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.471802 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.471774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-certs\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.472169 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.472143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836a83f6-4979-43ea-b001-72756d2de921-istiod-ca-cert\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.473724 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.473704 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-envoy\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.473825 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.473786 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836a83f6-4979-43ea-b001-72756d2de921-istio-podinfo\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.478713 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.478689 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-istio-token\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.478828 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.478689 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4ng\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-kube-api-access-lh4ng\") pod \"e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.486619 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.486597 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:55.626606 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.626584 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z"] Jul 28 14:57:55.628421 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:57:55.628397 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836a83f6_4979_43ea_b001_72756d2de921.slice/crio-959972c8c698d966b05e62bab30180185972244dd48db4c5d54386bef8df68ed WatchSource:0}: Error finding container 959972c8c698d966b05e62bab30180185972244dd48db4c5d54386bef8df68ed: Status 404 returned error can't find the container with id 959972c8c698d966b05e62bab30180185972244dd48db4c5d54386bef8df68ed Jul 28 14:57:55.630579 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.630546 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:57:55.630654 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.630615 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:57:55.630654 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:55.630644 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:57:56.303282 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:56.303223 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" event={"ID":"836a83f6-4979-43ea-b001-72756d2de921","Type":"ContainerStarted","Data":"6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e"} Jul 28 14:57:56.303496 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:56.303348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" event={"ID":"836a83f6-4979-43ea-b001-72756d2de921","Type":"ContainerStarted","Data":"959972c8c698d966b05e62bab30180185972244dd48db4c5d54386bef8df68ed"} Jul 28 14:57:56.323578 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:56.323524 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" podStartSLOduration=1.323508239 podStartE2EDuration="1.323508239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:57:56.321591268 +0000 UTC m=+2377.986833138" watchObservedRunningTime="2026-07-28 14:57:56.323508239 +0000 UTC m=+2377.988750061" Jul 28 14:57:56.486892 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:56.486867 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:56.491411 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:56.491385 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:56.964861 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:56.964824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:57.066005 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:57.065972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:57.309560 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:57.309532 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:57.310542 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:57.310518 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:57:58.273312 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:58.273275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:58.379209 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:58.379180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:58.505533 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:58.505504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:58.716263 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:58.716231 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:58.978746 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:58.978635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:57:59.063302 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:57:59.063272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:02.489373 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:02.489331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:58:10.385570 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:10.385532 2562 generic.go:356] "Generic (PLEG): container finished" podID="836a83f6-4979-43ea-b001-72756d2de921" containerID="6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e" exitCode=0 Jul 28 14:58:10.386030 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:10.385609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" event={"ID":"836a83f6-4979-43ea-b001-72756d2de921","Type":"ContainerDied","Data":"6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e"} Jul 28 14:58:10.386030 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:10.386014 2562 scope.go:117] "RemoveContainer" containerID="6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e" Jul 28 14:58:10.387135 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:10.387098 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:58:10.387196 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:10.387187 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:58:10.387236 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:10.387224 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 14:58:11.392814 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:11.392781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" event={"ID":"836a83f6-4979-43ea-b001-72756d2de921","Type":"ContainerStarted","Data":"458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c"} Jul 28 14:58:11.487240 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:11.487208 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:58:11.488243 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:11.488218 2562 patch_prober.go:28] interesting pod/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 28 14:58:11.488366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:11.488265 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 28 14:58:12.310061 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:12.310028 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:58:12.487588 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:12.487556 2562 patch_prober.go:28] interesting pod/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 28 14:58:12.487997 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:12.487604 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 28 14:58:13.487633 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:13.487598 2562 patch_prober.go:28] interesting pod/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 28 14:58:13.488052 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:13.487659 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 28 14:58:14.487175 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:14.487141 2562 patch_prober.go:28] interesting pod/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 28 14:58:14.487355 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:14.487195 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 28 14:58:15.490725 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:15.490694 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:58:15.491560 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:15.491542 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:58:19.019764 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:19.019735 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:58:19.029544 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:19.029525 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 14:58:23.333329 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:23.333277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:24.000366 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:24.000327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:24.104340 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:24.104313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:24.442691 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:24.442646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:30.254839 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.254794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:30.380804 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.380759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:30.634552 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.634525 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp"] Jul 28 14:58:30.638063 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.638045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:30.640323 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.640300 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 14:58:30.640460 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.640329 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 14:58:30.647728 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.647702 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp"] Jul 28 14:58:30.754067 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.754042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c639e41-35d2-476c-a627-619df1e04ceb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp\" (UID: \"8c639e41-35d2-476c-a627-619df1e04ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:30.855022 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.854989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c639e41-35d2-476c-a627-619df1e04ceb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp\" (UID: \"8c639e41-35d2-476c-a627-619df1e04ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:30.855559 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.855539 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c639e41-35d2-476c-a627-619df1e04ceb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp\" (UID: \"8c639e41-35d2-476c-a627-619df1e04ceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:30.949349 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:30.949289 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:31.088714 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:31.088688 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp"] Jul 28 14:58:31.090288 ip-10-0-131-121 kubenswrapper[2562]: W0728 14:58:31.090259 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c639e41_35d2_476c_a627_619df1e04ceb.slice/crio-ee2805fa71ef72cb686dd077f9febe52d1d2311c38bb80b1078bfd26fd80aa7e WatchSource:0}: Error finding container ee2805fa71ef72cb686dd077f9febe52d1d2311c38bb80b1078bfd26fd80aa7e: Status 404 returned error can't find the container with id ee2805fa71ef72cb686dd077f9febe52d1d2311c38bb80b1078bfd26fd80aa7e Jul 28 14:58:31.517293 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:31.517251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" event={"ID":"8c639e41-35d2-476c-a627-619df1e04ceb","Type":"ContainerStarted","Data":"9417ff3f6df0270ed25c00e6f7a27830e97ef639f4b06960049880b4965c3afa"} Jul 28 14:58:31.517625 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:31.517318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" event={"ID":"8c639e41-35d2-476c-a627-619df1e04ceb","Type":"ContainerStarted","Data":"ee2805fa71ef72cb686dd077f9febe52d1d2311c38bb80b1078bfd26fd80aa7e"} Jul 28 14:58:31.532954 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:31.532912 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" podStartSLOduration=1.532897162 podStartE2EDuration="1.532897162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:58:31.531384029 +0000 UTC m=+2413.196625869" watchObservedRunningTime="2026-07-28 14:58:31.532897162 +0000 UTC m=+2413.198139053" Jul 28 14:58:32.524865 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:32.524826 2562 generic.go:356] "Generic (PLEG): container finished" podID="8c639e41-35d2-476c-a627-619df1e04ceb" containerID="9417ff3f6df0270ed25c00e6f7a27830e97ef639f4b06960049880b4965c3afa" exitCode=0 Jul 28 14:58:32.525276 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:32.524912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" event={"ID":"8c639e41-35d2-476c-a627-619df1e04ceb","Type":"ContainerDied","Data":"9417ff3f6df0270ed25c00e6f7a27830e97ef639f4b06960049880b4965c3afa"} Jul 28 14:58:33.664666 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:33.664645 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:33.775991 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:33.775955 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c639e41-35d2-476c-a627-619df1e04ceb-maas-api-service-ca\") pod \"8c639e41-35d2-476c-a627-619df1e04ceb\" (UID: \"8c639e41-35d2-476c-a627-619df1e04ceb\") " Jul 28 14:58:33.776278 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:33.776254 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c639e41-35d2-476c-a627-619df1e04ceb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c639e41-35d2-476c-a627-619df1e04ceb" (UID: "8c639e41-35d2-476c-a627-619df1e04ceb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:58:33.877342 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:33.877316 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c639e41-35d2-476c-a627-619df1e04ceb-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:34.537996 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:34.537963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" event={"ID":"8c639e41-35d2-476c-a627-619df1e04ceb","Type":"ContainerDied","Data":"ee2805fa71ef72cb686dd077f9febe52d1d2311c38bb80b1078bfd26fd80aa7e"} Jul 28 14:58:34.537996 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:34.537992 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp" Jul 28 14:58:34.538195 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:34.538050 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2805fa71ef72cb686dd077f9febe52d1d2311c38bb80b1078bfd26fd80aa7e" Jul 28 14:58:39.749376 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:39.749316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:41.418955 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.418872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:41.481834 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.481801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:41.536840 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.536804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:41.577722 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.577668 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp"] Jul 28 14:58:41.580019 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.579999 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8b4faed4-g7nmp"] Jul 28 14:58:41.594389 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.594365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:41.977470 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:41.977441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:42.873967 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:42.873922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:42.890030 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:42.889997 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c639e41-35d2-476c-a627-619df1e04ceb" path="/var/lib/kubelet/pods/8c639e41-35d2-476c-a627-619df1e04ceb/volumes" Jul 28 14:58:42.924905 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:42.924874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:44.384367 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:44.384325 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:58:48.040559 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.040520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:48.114549 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.114516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:48.336492 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.336455 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z"] Jul 28 14:58:48.336727 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.336686 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" containerID="cri-o://458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c" gracePeriod=30 Jul 28 14:58:48.360622 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.360593 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:48.617592 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.617512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:48.752475 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.752444 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:48.849606 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:48.849582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:49.000972 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:49.000903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:53.584828 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.584808 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:58:53.650601 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.650532 2562 generic.go:356] "Generic (PLEG): container finished" podID="836a83f6-4979-43ea-b001-72756d2de921" containerID="458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c" exitCode=0 Jul 28 14:58:53.650601 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.650578 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" event={"ID":"836a83f6-4979-43ea-b001-72756d2de921","Type":"ContainerDied","Data":"458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c"} Jul 28 14:58:53.650601 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.650594 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" Jul 28 14:58:53.650866 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.650637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z" event={"ID":"836a83f6-4979-43ea-b001-72756d2de921","Type":"ContainerDied","Data":"959972c8c698d966b05e62bab30180185972244dd48db4c5d54386bef8df68ed"} Jul 28 14:58:53.650866 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.650658 2562 scope.go:117] "RemoveContainer" containerID="458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c" Jul 28 14:58:53.659554 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.659538 2562 scope.go:117] "RemoveContainer" containerID="6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e" Jul 28 14:58:53.667201 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.667185 2562 scope.go:117] "RemoveContainer" containerID="458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c" Jul 28 14:58:53.667408 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:58:53.667391 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c\": container with ID starting with 458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c not found: ID does not exist" containerID="458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c" Jul 28 14:58:53.667472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.667413 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c"} err="failed to get container status \"458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c\": rpc error: code = NotFound desc = could not find container \"458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c\": container with ID starting with 458fe50e84357c1efcc32292b883570819066da65d5f0b79e5fac8383709f94c not found: ID does not exist" Jul 28 14:58:53.667472 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.667427 2562 scope.go:117] "RemoveContainer" containerID="6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e" Jul 28 14:58:53.667635 ip-10-0-131-121 kubenswrapper[2562]: E0728 14:58:53.667620 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e\": container with ID starting with 6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e not found: ID does not exist" containerID="6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e" Jul 28 14:58:53.667700 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.667637 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e"} err="failed to get container status \"6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e\": rpc error: code = NotFound desc = could not find container \"6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e\": container with ID starting with 6f27331eea8f482ce724d35bc367109bab04fed8393c697dd843025264b62e5e not found: ID does not exist" Jul 28 14:58:53.741814 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.741790 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-credential-socket\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.741916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.741827 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-certs\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.741916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.741863 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-socket\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.741916 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.741886 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-envoy\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.742083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.741920 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh4ng\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-kube-api-access-lh4ng\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.742083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.741969 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836a83f6-4979-43ea-b001-72756d2de921-istiod-ca-cert\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.742083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742012 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-data\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.742083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742036 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-istio-token\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.742083 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742062 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836a83f6-4979-43ea-b001-72756d2de921-istio-podinfo\") pod \"836a83f6-4979-43ea-b001-72756d2de921\" (UID: \"836a83f6-4979-43ea-b001-72756d2de921\") " Jul 28 14:58:53.742315 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742060 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:58:53.742315 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742175 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:58:53.742315 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742200 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:58:53.742510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742359 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836a83f6-4979-43ea-b001-72756d2de921-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 14:58:53.742510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742406 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.742510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742428 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.742510 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.742443 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.743362 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.743331 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-data" (OuterVolumeSpecName: "istio-data") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:58:53.744107 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.744081 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 14:58:53.744191 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.744167 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-kube-api-access-lh4ng" (OuterVolumeSpecName: "kube-api-access-lh4ng") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "kube-api-access-lh4ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:58:53.744236 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.744185 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-istio-token" (OuterVolumeSpecName: "istio-token") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 14:58:53.744300 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.744283 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/836a83f6-4979-43ea-b001-72756d2de921-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "836a83f6-4979-43ea-b001-72756d2de921" (UID: "836a83f6-4979-43ea-b001-72756d2de921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 14:58:53.843299 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.843274 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4ng\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-kube-api-access-lh4ng\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.843299 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.843294 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836a83f6-4979-43ea-b001-72756d2de921-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.843415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.843309 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.843415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.843320 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836a83f6-4979-43ea-b001-72756d2de921-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.843415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.843328 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836a83f6-4979-43ea-b001-72756d2de921-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.843415 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.843336 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836a83f6-4979-43ea-b001-72756d2de921-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 14:58:53.974121 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.974089 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z"] Jul 28 14:58:53.980462 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:53.980434 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8b4faed4-openshift-default-767d4bb7b-dv45z"] Jul 28 14:58:54.889165 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:54.889133 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="836a83f6-4979-43ea-b001-72756d2de921" path="/var/lib/kubelet/pods/836a83f6-4979-43ea-b001-72756d2de921/volumes" Jul 28 14:58:54.896628 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:54.896603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:54.970180 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:54.970153 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:55.064073 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:55.064045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:55.205270 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:55.205200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:58:55.330569 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:58:55.330546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:01.212771 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:01.212726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:01.313094 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:01.313062 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:08.373652 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:08.373602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:08.742967 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:08.742892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:08.951563 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:08.951534 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:11.924999 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:11.924963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:12.005310 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:12.005275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:12.079627 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:12.079596 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:12.174259 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:12.174232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:12.543093 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:12.543052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:12.623001 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:12.622950 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:17.888256 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:17.888216 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 14:59:36.945310 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:36.945272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:43.690219 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:43.690184 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 14:59:47.097951 ip-10-0-131-121 kubenswrapper[2562]: I0728 14:59:47.097911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:00:00.131200 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131168 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv"] Jul 28 15:00:00.131747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131505 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" Jul 28 15:00:00.131747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131519 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" Jul 28 15:00:00.131747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131545 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c639e41-35d2-476c-a627-619df1e04ceb" containerName="revoke-keys" Jul 28 15:00:00.131747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131551 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c639e41-35d2-476c-a627-619df1e04ceb" containerName="revoke-keys" Jul 28 15:00:00.131747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131656 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" Jul 28 15:00:00.131747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.131692 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c639e41-35d2-476c-a627-619df1e04ceb" containerName="revoke-keys" Jul 28 15:00:00.134797 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.134777 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk"] Jul 28 15:00:00.134922 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.134903 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" Jul 28 15:00:00.135088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.135067 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" Jul 28 15:00:00.135088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.135079 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" Jul 28 15:00:00.135207 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.135178 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="836a83f6-4979-43ea-b001-72756d2de921" containerName="istio-proxy" Jul 28 15:00:00.137187 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.137167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:00:00.137976 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.137957 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" Jul 28 15:00:00.143582 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.143558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv"] Jul 28 15:00:00.147153 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.147131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk"] Jul 28 15:00:00.148150 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.148129 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" Jul 28 15:00:00.156393 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.156375 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" Jul 28 15:00:00.313727 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.313702 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk"] Jul 28 15:00:00.315505 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:00:00.315474 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336aa749_62e1_4351_b83b_c61dc378c8dd.slice/crio-49081c03a892217b66fa8a9d0a2b5809a4091412c4aa60ae4bfe1c740a9ed41d WatchSource:0}: Error finding container 49081c03a892217b66fa8a9d0a2b5809a4091412c4aa60ae4bfe1c740a9ed41d: Status 404 returned error can't find the container with id 49081c03a892217b66fa8a9d0a2b5809a4091412c4aa60ae4bfe1c740a9ed41d Jul 28 15:00:00.496598 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:00.496571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv"] Jul 28 15:00:00.497906 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:00:00.497878 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca85b39_28ac_4ad8_9a1e_28c89cef4c4a.slice/crio-c430d983e06e67922ab99d61b10ef788511debdaf4c46617523d659fe4b2addd WatchSource:0}: Error finding container c430d983e06e67922ab99d61b10ef788511debdaf4c46617523d659fe4b2addd: Status 404 returned error can't find the container with id c430d983e06e67922ab99d61b10ef788511debdaf4c46617523d659fe4b2addd Jul 28 15:00:01.020586 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.020510 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" event={"ID":"336aa749-62e1-4351-b83b-c61dc378c8dd","Type":"ContainerStarted","Data":"f2ace44adc522024a413e91785bdd3aa61797adba56d5a4b621ba95373342169"} Jul 28 15:00:01.020586 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.020574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" event={"ID":"336aa749-62e1-4351-b83b-c61dc378c8dd","Type":"ContainerStarted","Data":"49081c03a892217b66fa8a9d0a2b5809a4091412c4aa60ae4bfe1c740a9ed41d"} Jul 28 15:00:01.021944 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.021916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" event={"ID":"fca85b39-28ac-4ad8-9a1e-28c89cef4c4a","Type":"ContainerStarted","Data":"7985acd0379b88260409af7520c5e3418dc54ab4a6340d951ff3b4085b57d9ac"} Jul 28 15:00:01.022059 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.021969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" event={"ID":"fca85b39-28ac-4ad8-9a1e-28c89cef4c4a","Type":"ContainerStarted","Data":"c430d983e06e67922ab99d61b10ef788511debdaf4c46617523d659fe4b2addd"} Jul 28 15:00:01.034664 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.034622 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" podStartSLOduration=1.034607269 podStartE2EDuration="1.034607269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:00:01.033277563 +0000 UTC m=+2502.698519398" watchObservedRunningTime="2026-07-28 15:00:01.034607269 +0000 UTC m=+2502.699849070" Jul 28 15:00:01.051577 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.051534 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" podStartSLOduration=1.05152066 podStartE2EDuration="1.05152066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:00:01.048950887 +0000 UTC m=+2502.714192711" watchObservedRunningTime="2026-07-28 15:00:01.05152066 +0000 UTC m=+2502.716762482" Jul 28 15:00:01.687549 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:01.687486 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:00:02.028922 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:02.028838 2562 generic.go:356] "Generic (PLEG): container finished" podID="fca85b39-28ac-4ad8-9a1e-28c89cef4c4a" containerID="7985acd0379b88260409af7520c5e3418dc54ab4a6340d951ff3b4085b57d9ac" exitCode=0 Jul 28 15:00:02.029064 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:02.028925 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" event={"ID":"fca85b39-28ac-4ad8-9a1e-28c89cef4c4a","Type":"ContainerDied","Data":"7985acd0379b88260409af7520c5e3418dc54ab4a6340d951ff3b4085b57d9ac"} Jul 28 15:00:02.030473 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:02.030448 2562 generic.go:356] "Generic (PLEG): container finished" podID="336aa749-62e1-4351-b83b-c61dc378c8dd" containerID="f2ace44adc522024a413e91785bdd3aa61797adba56d5a4b621ba95373342169" exitCode=0 Jul 28 15:00:02.030578 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:02.030533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" event={"ID":"336aa749-62e1-4351-b83b-c61dc378c8dd","Type":"ContainerDied","Data":"f2ace44adc522024a413e91785bdd3aa61797adba56d5a4b621ba95373342169"} Jul 28 15:00:03.177393 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:03.177363 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" Jul 28 15:00:03.204282 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:03.204263 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" Jul 28 15:00:04.042595 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:04.042567 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" Jul 28 15:00:04.042802 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:04.042602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754180-5twkk" event={"ID":"336aa749-62e1-4351-b83b-c61dc378c8dd","Type":"ContainerDied","Data":"49081c03a892217b66fa8a9d0a2b5809a4091412c4aa60ae4bfe1c740a9ed41d"} Jul 28 15:00:04.042802 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:04.042654 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49081c03a892217b66fa8a9d0a2b5809a4091412c4aa60ae4bfe1c740a9ed41d" Jul 28 15:00:04.044015 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:04.043985 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" Jul 28 15:00:04.044174 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:04.044014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv" event={"ID":"fca85b39-28ac-4ad8-9a1e-28c89cef4c4a","Type":"ContainerDied","Data":"c430d983e06e67922ab99d61b10ef788511debdaf4c46617523d659fe4b2addd"} Jul 28 15:00:04.044174 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:04.044050 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c430d983e06e67922ab99d61b10ef788511debdaf4c46617523d659fe4b2addd" Jul 28 15:00:06.253043 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253012 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt"] Jul 28 15:00:06.253421 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253354 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336aa749-62e1-4351-b83b-c61dc378c8dd" containerName="cleanup" Jul 28 15:00:06.253421 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253366 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="336aa749-62e1-4351-b83b-c61dc378c8dd" containerName="cleanup" Jul 28 15:00:06.253421 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253385 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fca85b39-28ac-4ad8-9a1e-28c89cef4c4a" containerName="cleanup" Jul 28 15:00:06.253421 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253391 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fca85b39-28ac-4ad8-9a1e-28c89cef4c4a" containerName="cleanup" Jul 28 15:00:06.253560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253446 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fca85b39-28ac-4ad8-9a1e-28c89cef4c4a" containerName="cleanup" Jul 28 15:00:06.253560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.253456 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="336aa749-62e1-4351-b83b-c61dc378c8dd" containerName="cleanup" Jul 28 15:00:06.256473 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.256455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:06.258565 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.258544 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 15:00:06.258701 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.258550 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:00:06.263964 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.263944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt"] Jul 28 15:00:06.381336 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.381301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57962672-f425-44fa-8cee-453278825369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt\" (UID: \"57962672-f425-44fa-8cee-453278825369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:06.482248 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.482218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57962672-f425-44fa-8cee-453278825369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt\" (UID: \"57962672-f425-44fa-8cee-453278825369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:06.482872 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.482851 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57962672-f425-44fa-8cee-453278825369-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt\" (UID: \"57962672-f425-44fa-8cee-453278825369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:06.568352 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.568326 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:06.696892 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:06.696865 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt"] Jul 28 15:00:06.699049 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:00:06.699022 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57962672_f425_44fa_8cee_453278825369.slice/crio-9b541c9b764c4b94581193ef3f547cdc5a0822fe5aca4fbd56d8095050e4768c WatchSource:0}: Error finding container 9b541c9b764c4b94581193ef3f547cdc5a0822fe5aca4fbd56d8095050e4768c: Status 404 returned error can't find the container with id 9b541c9b764c4b94581193ef3f547cdc5a0822fe5aca4fbd56d8095050e4768c Jul 28 15:00:07.060321 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:07.060294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" event={"ID":"57962672-f425-44fa-8cee-453278825369","Type":"ContainerStarted","Data":"f7c200c4f22f8e2aea9cffdb0d4680a0b845749914435ce0fedb8fdf0bf64aec"} Jul 28 15:00:07.060555 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:07.060341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" event={"ID":"57962672-f425-44fa-8cee-453278825369","Type":"ContainerStarted","Data":"9b541c9b764c4b94581193ef3f547cdc5a0822fe5aca4fbd56d8095050e4768c"} Jul 28 15:00:07.075620 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:07.075575 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" podStartSLOduration=1.075560339 podStartE2EDuration="1.075560339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:00:07.07427497 +0000 UTC m=+2508.739516795" watchObservedRunningTime="2026-07-28 15:00:07.075560339 +0000 UTC m=+2508.740802154" Jul 28 15:00:08.067818 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:08.067788 2562 generic.go:356] "Generic (PLEG): container finished" podID="57962672-f425-44fa-8cee-453278825369" containerID="f7c200c4f22f8e2aea9cffdb0d4680a0b845749914435ce0fedb8fdf0bf64aec" exitCode=0 Jul 28 15:00:08.068274 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:08.067875 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" event={"ID":"57962672-f425-44fa-8cee-453278825369","Type":"ContainerDied","Data":"f7c200c4f22f8e2aea9cffdb0d4680a0b845749914435ce0fedb8fdf0bf64aec"} Jul 28 15:00:09.206721 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:09.206695 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:09.303637 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:09.303614 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57962672-f425-44fa-8cee-453278825369-maas-api-service-ca\") pod \"57962672-f425-44fa-8cee-453278825369\" (UID: \"57962672-f425-44fa-8cee-453278825369\") " Jul 28 15:00:09.303949 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:09.303926 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57962672-f425-44fa-8cee-453278825369-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57962672-f425-44fa-8cee-453278825369" (UID: "57962672-f425-44fa-8cee-453278825369"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:00:09.404284 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:09.404230 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57962672-f425-44fa-8cee-453278825369-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:00:10.080192 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:10.080171 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" Jul 28 15:00:10.080343 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:10.080208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt" event={"ID":"57962672-f425-44fa-8cee-453278825369","Type":"ContainerDied","Data":"9b541c9b764c4b94581193ef3f547cdc5a0822fe5aca4fbd56d8095050e4768c"} Jul 28 15:00:10.080343 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:10.080254 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b541c9b764c4b94581193ef3f547cdc5a0822fe5aca4fbd56d8095050e4768c" Jul 28 15:00:14.923371 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:14.923321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:14.924023 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:14.923995 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv"] Jul 28 15:00:14.926437 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:14.926413 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-917bdcc4-29754180-sv5cv"] Jul 28 15:00:14.996578 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:14.996545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:15.097410 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:15.097365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:15.098166 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:15.098140 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt"] Jul 28 15:00:15.100284 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:15.100264 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-917bdcc4-fvcqt"] Jul 28 15:00:15.196652 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:15.196572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:16.368856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:16.368820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:16.889033 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:16.888992 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57962672-f425-44fa-8cee-453278825369" path="/var/lib/kubelet/pods/57962672-f425-44fa-8cee-453278825369/volumes" Jul 28 15:00:16.889365 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:16.889345 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fca85b39-28ac-4ad8-9a1e-28c89cef4c4a" path="/var/lib/kubelet/pods/fca85b39-28ac-4ad8-9a1e-28c89cef4c4a/volumes" Jul 28 15:00:17.495016 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:17.494975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:00:19.263149 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:19.263119 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:20.515371 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:20.515336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:20.580552 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:20.580517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:23.429592 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:23.429552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:00:33.141750 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:00:33.141711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:00.882529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:00.882494 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:01.498449 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:01.498397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:07.248485 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:07.248440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:33.153188 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:33.153145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:35.073149 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:35.073111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:35.272835 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:35.272804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:35.397733 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:35.397643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:36.435203 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:36.435165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:01:36.624827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:01:36.624789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:03:19.064366 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:03:19.064296 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:03:19.071848 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:03:19.071828 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:03:56.935729 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:03:56.935660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:05:12.582030 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:05:12.581991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:05:18.506433 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:05:18.506391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:05:19.178828 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:05:19.178793 2562 scope.go:117] "RemoveContainer" containerID="9417ff3f6df0270ed25c00e6f7a27830e97ef639f4b06960049880b4965c3afa" Jul 28 15:06:19.198937 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:19.198852 2562 scope.go:117] "RemoveContainer" containerID="7985acd0379b88260409af7520c5e3418dc54ab4a6340d951ff3b4085b57d9ac" Jul 28 15:06:19.207117 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:19.207098 2562 scope.go:117] "RemoveContainer" containerID="f7c200c4f22f8e2aea9cffdb0d4680a0b845749914435ce0fedb8fdf0bf64aec" Jul 28 15:06:43.295231 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:43.295196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:06:50.072167 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:50.072137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:06:50.138932 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:50.138908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:06:50.252330 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:50.252307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:06:50.338364 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:50.338300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:06:51.680355 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:51.680321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:06:51.750598 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:06:51.750568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:00.872317 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:00.872284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:01.231980 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:01.231882 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:09.000800 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:09.000765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:22.210598 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:22.210561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:28.915762 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:28.915732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:29.040398 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.040367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:29.289951 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.289874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:29.579032 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.579001 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc"] Jul 28 15:07:29.579401 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.579384 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57962672-f425-44fa-8cee-453278825369" containerName="revoke-keys" Jul 28 15:07:29.579455 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.579405 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="57962672-f425-44fa-8cee-453278825369" containerName="revoke-keys" Jul 28 15:07:29.579492 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.579474 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="57962672-f425-44fa-8cee-453278825369" containerName="revoke-keys" Jul 28 15:07:29.582439 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.582421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.584640 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.584622 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 15:07:29.585044 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.585023 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-06cd69-openshift-default-dockercfg-ndnrt\"" Jul 28 15:07:29.600182 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.600157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:29.606529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.606505 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc"] Jul 28 15:07:29.719919 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.719886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-certs\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.719931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-credential-socket\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.719971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f071-f672-47b0-977d-6159b501fa99-istiod-ca-cert\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.720013 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-istio-token\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.720050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-data\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.720081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f071-f672-47b0-977d-6159b501fa99-istio-podinfo\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.720100 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-envoy\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720316 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.720170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-socket\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.720316 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.720196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qbp\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-kube-api-access-l7qbp\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.760423 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.760390 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:29.821274 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-credential-socket\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f071-f672-47b0-977d-6159b501fa99-istiod-ca-cert\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-istio-token\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-data\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821354 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f071-f672-47b0-977d-6159b501fa99-istio-podinfo\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-envoy\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-socket\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821440 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qbp\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-kube-api-access-l7qbp\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821757 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-certs\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.821971 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.821947 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-certs\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.822118 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.822023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-credential-socket\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.822185 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.822117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-data\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.822185 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.822158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-socket\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.822273 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.822166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f071-f672-47b0-977d-6159b501fa99-istiod-ca-cert\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.824043 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.824020 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-envoy\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.824133 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.824080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f071-f672-47b0-977d-6159b501fa99-istio-podinfo\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.829563 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.829500 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-istio-token\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.829819 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.829801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qbp\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-kube-api-access-l7qbp\") pod \"e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:29.892627 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:29.892601 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:30.036739 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.036708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc"] Jul 28 15:07:30.038748 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:07:30.038724 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb2f071_f672_47b0_977d_6159b501fa99.slice/crio-84965fcaf012531560db45d57249b6aca76edfe8f407829dd72c4104996e9ac2 WatchSource:0}: Error finding container 84965fcaf012531560db45d57249b6aca76edfe8f407829dd72c4104996e9ac2: Status 404 returned error can't find the container with id 84965fcaf012531560db45d57249b6aca76edfe8f407829dd72c4104996e9ac2 Jul 28 15:07:30.041001 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.040969 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:07:30.041075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.041038 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:07:30.041075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.041069 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:07:30.504341 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.504256 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" event={"ID":"5fb2f071-f672-47b0-977d-6159b501fa99","Type":"ContainerStarted","Data":"52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e"} Jul 28 15:07:30.504341 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.504312 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" event={"ID":"5fb2f071-f672-47b0-977d-6159b501fa99","Type":"ContainerStarted","Data":"84965fcaf012531560db45d57249b6aca76edfe8f407829dd72c4104996e9ac2"} Jul 28 15:07:30.525468 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.525410 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" podStartSLOduration=1.525391118 podStartE2EDuration="1.525391118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:07:30.522319931 +0000 UTC m=+2952.187561759" watchObservedRunningTime="2026-07-28 15:07:30.525391118 +0000 UTC m=+2952.190632921" Jul 28 15:07:30.893326 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:30.893300 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:31.898940 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:31.898907 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:32.466865 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:32.466835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:32.515533 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:32.515508 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:32.516370 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:32.516350 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:32.559213 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:32.559164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:32.649087 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:32.649050 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:32.761860 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:32.761784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:33.154497 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:33.154467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:33.243256 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:33.243224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:37.884844 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:37.884808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:07:45.585491 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:45.585430 2562 generic.go:356] "Generic (PLEG): container finished" podID="5fb2f071-f672-47b0-977d-6159b501fa99" containerID="52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e" exitCode=0 Jul 28 15:07:45.585491 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:45.585467 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" event={"ID":"5fb2f071-f672-47b0-977d-6159b501fa99","Type":"ContainerDied","Data":"52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e"} Jul 28 15:07:45.585914 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:45.585897 2562 scope.go:117] "RemoveContainer" containerID="52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e" Jul 28 15:07:45.586973 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:45.586941 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:07:45.587088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:45.587026 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:07:45.587151 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:45.587082 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:07:46.592279 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:46.592246 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" event={"ID":"5fb2f071-f672-47b0-977d-6159b501fa99","Type":"ContainerStarted","Data":"9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5"} Jul 28 15:07:46.893711 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:46.893602 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:46.894866 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:46.894842 2562 patch_prober.go:28] interesting pod/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 28 15:07:46.894999 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:46.894885 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 28 15:07:47.515474 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:47.515435 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:47.893328 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:47.893287 2562 patch_prober.go:28] interesting pod/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 28 15:07:47.893773 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:47.893349 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 28 15:07:48.894020 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:48.893982 2562 patch_prober.go:28] interesting pod/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 28 15:07:48.894381 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:48.894051 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 28 15:07:49.893093 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:49.893049 2562 patch_prober.go:28] interesting pod/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 28 15:07:49.893315 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:49.893135 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 28 15:07:50.896619 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:50.896592 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:50.897521 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:50.897502 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:07:54.158640 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:54.158602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:54.223856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:54.223819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:57.485379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:57.485345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:57.836966 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:57.836930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:58.185609 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:58.185514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:07:58.353279 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:07:58.353246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:01.401055 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:01.401015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:01.506401 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:01.506368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:01.720794 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:01.720704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:01.861573 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:01.861540 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:02.182581 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:02.182546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:02.280919 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:02.280880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:08.596785 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:08.596742 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:08:19.099894 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:19.099864 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:08:19.108470 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:19.108444 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:08:26.441184 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:26.441138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:33.206987 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:33.206948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:08:37.400490 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:37.400453 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:08:48.992783 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:08:48.992746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:09:18.270342 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:09:18.270264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:09:23.515801 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:09:23.515765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:10:02.805689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:02.805631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:02.957128 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:02.957057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:22.296030 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:22.295997 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:23.225135 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.225104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:23.339272 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.339236 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz"] Jul 28 15:10:23.343029 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.343008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:23.345411 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.345388 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 15:10:23.345566 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.345541 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:10:23.353223 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.353197 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz"] Jul 28 15:10:23.461235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.461198 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978056a6-2966-4a69-bf20-cb9aa67782fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz\" (UID: \"978056a6-2966-4a69-bf20-cb9aa67782fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:23.561753 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.561719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978056a6-2966-4a69-bf20-cb9aa67782fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz\" (UID: \"978056a6-2966-4a69-bf20-cb9aa67782fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:23.562322 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.562302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978056a6-2966-4a69-bf20-cb9aa67782fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz\" (UID: \"978056a6-2966-4a69-bf20-cb9aa67782fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:23.653598 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.653563 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:23.995927 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:23.995846 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz"] Jul 28 15:10:23.999473 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:10:23.999446 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978056a6_2966_4a69_bf20_cb9aa67782fa.slice/crio-3cfa5ac811dc0b1eeab1d5a1354548e859c88a4ad209e8ee4660c47897460845 WatchSource:0}: Error finding container 3cfa5ac811dc0b1eeab1d5a1354548e859c88a4ad209e8ee4660c47897460845: Status 404 returned error can't find the container with id 3cfa5ac811dc0b1eeab1d5a1354548e859c88a4ad209e8ee4660c47897460845 Jul 28 15:10:24.442845 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:24.442811 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" event={"ID":"978056a6-2966-4a69-bf20-cb9aa67782fa","Type":"ContainerStarted","Data":"2d187682b88582d2fdd68c755c9bb699ee58e01533e84242f1d783b452286171"} Jul 28 15:10:24.443247 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:24.442868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" event={"ID":"978056a6-2966-4a69-bf20-cb9aa67782fa","Type":"ContainerStarted","Data":"3cfa5ac811dc0b1eeab1d5a1354548e859c88a4ad209e8ee4660c47897460845"} Jul 28 15:10:24.460298 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:24.460248 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" podStartSLOduration=1.460231512 podStartE2EDuration="1.460231512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:10:24.458500001 +0000 UTC m=+3126.123741827" watchObservedRunningTime="2026-07-28 15:10:24.460231512 +0000 UTC m=+3126.125473334" Jul 28 15:10:25.449798 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:25.449757 2562 generic.go:356] "Generic (PLEG): container finished" podID="978056a6-2966-4a69-bf20-cb9aa67782fa" containerID="2d187682b88582d2fdd68c755c9bb699ee58e01533e84242f1d783b452286171" exitCode=0 Jul 28 15:10:25.450257 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:25.449830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" event={"ID":"978056a6-2966-4a69-bf20-cb9aa67782fa","Type":"ContainerDied","Data":"2d187682b88582d2fdd68c755c9bb699ee58e01533e84242f1d783b452286171"} Jul 28 15:10:26.592388 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.592357 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:26.690038 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.689995 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978056a6-2966-4a69-bf20-cb9aa67782fa-maas-api-service-ca\") pod \"978056a6-2966-4a69-bf20-cb9aa67782fa\" (UID: \"978056a6-2966-4a69-bf20-cb9aa67782fa\") " Jul 28 15:10:26.690362 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.690343 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978056a6-2966-4a69-bf20-cb9aa67782fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "978056a6-2966-4a69-bf20-cb9aa67782fa" (UID: "978056a6-2966-4a69-bf20-cb9aa67782fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:10:26.790905 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.790814 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978056a6-2966-4a69-bf20-cb9aa67782fa-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:26.797854 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.797827 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:10:26.831934 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.831906 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vsdrp"] Jul 28 15:10:26.832309 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.832294 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978056a6-2966-4a69-bf20-cb9aa67782fa" containerName="revoke-keys" Jul 28 15:10:26.832309 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.832310 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="978056a6-2966-4a69-bf20-cb9aa67782fa" containerName="revoke-keys" Jul 28 15:10:26.832416 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.832374 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="978056a6-2966-4a69-bf20-cb9aa67782fa" containerName="revoke-keys" Jul 28 15:10:26.835607 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.835590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:26.838085 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.838066 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-f5lnh\"" Jul 28 15:10:26.845610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.845589 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vsdrp"] Jul 28 15:10:26.992398 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:26.992343 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbvv\" (UniqueName: \"kubernetes.io/projected/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5-kube-api-access-5wbvv\") pod \"kuadrant-operator-catalog-vsdrp\" (UID: \"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5\") " pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:27.093485 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.093444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbvv\" (UniqueName: \"kubernetes.io/projected/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5-kube-api-access-5wbvv\") pod \"kuadrant-operator-catalog-vsdrp\" (UID: \"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5\") " pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:27.102845 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.102821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbvv\" (UniqueName: \"kubernetes.io/projected/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5-kube-api-access-5wbvv\") pod \"kuadrant-operator-catalog-vsdrp\" (UID: \"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5\") " pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:27.145145 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.145108 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:27.281249 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.281216 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vsdrp"] Jul 28 15:10:27.283025 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:10:27.282994 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774e6ecb_bf34_4c59_9fc2_f28e9f8a8be5.slice/crio-fcc920abe64c16d331d45aff112142aaf415e8d2c65b9f013d86cc4f50f7ee31 WatchSource:0}: Error finding container fcc920abe64c16d331d45aff112142aaf415e8d2c65b9f013d86cc4f50f7ee31: Status 404 returned error can't find the container with id fcc920abe64c16d331d45aff112142aaf415e8d2c65b9f013d86cc4f50f7ee31 Jul 28 15:10:27.284571 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.284555 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:10:27.462529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.462443 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" Jul 28 15:10:27.462529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.462479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz" event={"ID":"978056a6-2966-4a69-bf20-cb9aa67782fa","Type":"ContainerDied","Data":"3cfa5ac811dc0b1eeab1d5a1354548e859c88a4ad209e8ee4660c47897460845"} Jul 28 15:10:27.462529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.462527 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfa5ac811dc0b1eeab1d5a1354548e859c88a4ad209e8ee4660c47897460845" Jul 28 15:10:27.463641 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:27.463613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" event={"ID":"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5","Type":"ContainerStarted","Data":"fcc920abe64c16d331d45aff112142aaf415e8d2c65b9f013d86cc4f50f7ee31"} Jul 28 15:10:30.481689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:30.481645 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" event={"ID":"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5","Type":"ContainerStarted","Data":"640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6"} Jul 28 15:10:30.502858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:30.502813 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" podStartSLOduration=2.368190872 podStartE2EDuration="4.50279983s" totalImagesPullingTime="2.134608958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:10:30.499850327 +0000 UTC m=+3132.165092151" watchObservedRunningTime="2026-07-28 15:10:30.50279983 +0000 UTC m=+3132.168041653" Jul 28 15:10:34.077871 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.077837 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:34.181232 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.181199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:34.291858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.291823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:34.323747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.323720 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz"] Jul 28 15:10:34.330592 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.330536 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06cd69-p64kz"] Jul 28 15:10:34.778834 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.778754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:34.888450 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:34.888417 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978056a6-2966-4a69-bf20-cb9aa67782fa" path="/var/lib/kubelet/pods/978056a6-2966-4a69-bf20-cb9aa67782fa/volumes" Jul 28 15:10:37.145392 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:37.145359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:37.145824 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:37.145419 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:37.167423 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:37.167396 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:37.543069 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:37.542999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:37.612282 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:37.612252 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vsdrp"] Jul 28 15:10:39.532833 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:39.532796 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" podUID="774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" containerName="registry-server" containerID="cri-o://640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6" gracePeriod=2 Jul 28 15:10:39.777257 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:39.777230 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:39.906028 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:39.905998 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wbvv\" (UniqueName: \"kubernetes.io/projected/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5-kube-api-access-5wbvv\") pod \"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5\" (UID: \"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5\") " Jul 28 15:10:39.908061 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:39.908032 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5-kube-api-access-5wbvv" (OuterVolumeSpecName: "kube-api-access-5wbvv") pod "774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" (UID: "774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5"). InnerVolumeSpecName "kube-api-access-5wbvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:10:40.007180 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.007127 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbvv\" (UniqueName: \"kubernetes.io/projected/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5-kube-api-access-5wbvv\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:40.539857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.539772 2562 generic.go:356] "Generic (PLEG): container finished" podID="774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" containerID="640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6" exitCode=0 Jul 28 15:10:40.539857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.539824 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" Jul 28 15:10:40.540327 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.539858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" event={"ID":"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5","Type":"ContainerDied","Data":"640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6"} Jul 28 15:10:40.540327 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.539916 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vsdrp" event={"ID":"774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5","Type":"ContainerDied","Data":"fcc920abe64c16d331d45aff112142aaf415e8d2c65b9f013d86cc4f50f7ee31"} Jul 28 15:10:40.540327 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.539939 2562 scope.go:117] "RemoveContainer" containerID="640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6" Jul 28 15:10:40.550719 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.550700 2562 scope.go:117] "RemoveContainer" containerID="640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6" Jul 28 15:10:40.550994 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:10:40.550974 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6\": container with ID starting with 640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6 not found: ID does not exist" containerID="640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6" Jul 28 15:10:40.551070 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.551002 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6"} err="failed to get container status \"640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6\": rpc error: code = NotFound desc = could not find container \"640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6\": container with ID starting with 640ab254c6dc965179e40f3076c3921c0416cf1635093fdcbb6f969aa8320bb6 not found: ID does not exist" Jul 28 15:10:40.571782 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.571755 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vsdrp"] Jul 28 15:10:40.576235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.576212 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vsdrp"] Jul 28 15:10:40.821472 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.821436 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:10:40.889180 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:40.889148 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" path="/var/lib/kubelet/pods/774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5/volumes" Jul 28 15:10:42.262129 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.262095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:42.364345 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.364300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:42.475102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.475071 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc"] Jul 28 15:10:42.475313 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.475291 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" containerID="cri-o://9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5" gracePeriod=30 Jul 28 15:10:42.543393 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.543313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:42.872785 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.872749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:42.920946 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.920914 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6"] Jul 28 15:10:42.921290 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.921276 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" containerName="registry-server" Jul 28 15:10:42.921335 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.921292 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" containerName="registry-server" Jul 28 15:10:42.921374 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.921355 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="774e6ecb-bf34-4c59-9fc2-f28e9f8a8be5" containerName="registry-server" Jul 28 15:10:42.925800 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.925782 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:42.930758 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.930739 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:10:42.936084 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.936063 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 15:10:42.955888 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:42.955862 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6"] Jul 28 15:10:43.035105 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:43.035065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b68c2f-b52b-4327-a03d-2ba5d7905244-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6\" (UID: \"45b68c2f-b52b-4327-a03d-2ba5d7905244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:43.136124 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:43.136017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b68c2f-b52b-4327-a03d-2ba5d7905244-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6\" (UID: \"45b68c2f-b52b-4327-a03d-2ba5d7905244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:43.136814 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:43.136791 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b68c2f-b52b-4327-a03d-2ba5d7905244-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6\" (UID: \"45b68c2f-b52b-4327-a03d-2ba5d7905244\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:43.235961 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:43.235926 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:43.391484 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:43.391457 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6"] Jul 28 15:10:43.393162 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:10:43.393134 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b68c2f_b52b_4327_a03d_2ba5d7905244.slice/crio-1d4235ad876da038a314111d3a51d4eaf007789fef4486217a89e9dff51c00b9 WatchSource:0}: Error finding container 1d4235ad876da038a314111d3a51d4eaf007789fef4486217a89e9dff51c00b9: Status 404 returned error can't find the container with id 1d4235ad876da038a314111d3a51d4eaf007789fef4486217a89e9dff51c00b9 Jul 28 15:10:43.558431 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:43.558396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" event={"ID":"45b68c2f-b52b-4327-a03d-2ba5d7905244","Type":"ContainerStarted","Data":"1d4235ad876da038a314111d3a51d4eaf007789fef4486217a89e9dff51c00b9"} Jul 28 15:10:44.565649 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:44.565614 2562 generic.go:356] "Generic (PLEG): container finished" podID="45b68c2f-b52b-4327-a03d-2ba5d7905244" containerID="a0c2e9d41219adef3b9637d54f4b92724d6870b10c64479e0d80d96ee0911551" exitCode=0 Jul 28 15:10:44.566023 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:44.565702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" event={"ID":"45b68c2f-b52b-4327-a03d-2ba5d7905244","Type":"ContainerDied","Data":"a0c2e9d41219adef3b9637d54f4b92724d6870b10c64479e0d80d96ee0911551"} Jul 28 15:10:45.705970 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:45.705948 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:45.758163 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:45.758135 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b68c2f-b52b-4327-a03d-2ba5d7905244-maas-api-service-ca\") pod \"45b68c2f-b52b-4327-a03d-2ba5d7905244\" (UID: \"45b68c2f-b52b-4327-a03d-2ba5d7905244\") " Jul 28 15:10:45.758467 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:45.758446 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b68c2f-b52b-4327-a03d-2ba5d7905244-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45b68c2f-b52b-4327-a03d-2ba5d7905244" (UID: "45b68c2f-b52b-4327-a03d-2ba5d7905244"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:10:45.859438 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:45.859382 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b68c2f-b52b-4327-a03d-2ba5d7905244-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:46.578433 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:46.578408 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" Jul 28 15:10:46.578594 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:46.578452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6" event={"ID":"45b68c2f-b52b-4327-a03d-2ba5d7905244","Type":"ContainerDied","Data":"1d4235ad876da038a314111d3a51d4eaf007789fef4486217a89e9dff51c00b9"} Jul 28 15:10:46.578594 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:46.578505 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d4235ad876da038a314111d3a51d4eaf007789fef4486217a89e9dff51c00b9" Jul 28 15:10:47.744327 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.744279 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:10:47.771392 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771367 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-istio-token\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771492 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771403 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-envoy\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771492 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771428 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-socket\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771492 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771466 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f071-f672-47b0-977d-6159b501fa99-istiod-ca-cert\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771492 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-data\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771714 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771516 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f071-f672-47b0-977d-6159b501fa99-istio-podinfo\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771714 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771536 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-certs\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771839 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771762 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-credential-socket\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.771839 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771776 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:10:47.771839 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771797 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7qbp\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-kube-api-access-l7qbp\") pod \"5fb2f071-f672-47b0-977d-6159b501fa99\" (UID: \"5fb2f071-f672-47b0-977d-6159b501fa99\") " Jul 28 15:10:47.772005 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.771843 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb2f071-f672-47b0-977d-6159b501fa99-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:10:47.772062 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.772019 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:10:47.772175 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.772144 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:10:47.772175 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.772153 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.772290 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.772185 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f071-f672-47b0-977d-6159b501fa99-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.772290 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.772200 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.772404 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.772388 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-data" (OuterVolumeSpecName: "istio-data") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:10:47.773930 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.773905 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-istio-token" (OuterVolumeSpecName: "istio-token") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:10:47.773930 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.773916 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb2f071-f672-47b0-977d-6159b501fa99-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 15:10:47.774043 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.774000 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:10:47.774082 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.774043 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-kube-api-access-l7qbp" (OuterVolumeSpecName: "kube-api-access-l7qbp") pod "5fb2f071-f672-47b0-977d-6159b501fa99" (UID: "5fb2f071-f672-47b0-977d-6159b501fa99"). InnerVolumeSpecName "kube-api-access-l7qbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:10:47.872586 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.872557 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.872586 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.872581 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f071-f672-47b0-977d-6159b501fa99-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.872771 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.872595 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.872771 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.872610 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qbp\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-kube-api-access-l7qbp\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.872771 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.872623 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f071-f672-47b0-977d-6159b501fa99-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:47.872771 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:47.872634 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f071-f672-47b0-977d-6159b501fa99-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:10:48.592103 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.592068 2562 generic.go:356] "Generic (PLEG): container finished" podID="5fb2f071-f672-47b0-977d-6159b501fa99" containerID="9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5" exitCode=0 Jul 28 15:10:48.592265 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.592119 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" Jul 28 15:10:48.592265 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.592156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" event={"ID":"5fb2f071-f672-47b0-977d-6159b501fa99","Type":"ContainerDied","Data":"9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5"} Jul 28 15:10:48.592265 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.592211 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc" event={"ID":"5fb2f071-f672-47b0-977d-6159b501fa99","Type":"ContainerDied","Data":"84965fcaf012531560db45d57249b6aca76edfe8f407829dd72c4104996e9ac2"} Jul 28 15:10:48.592265 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.592235 2562 scope.go:117] "RemoveContainer" containerID="9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5" Jul 28 15:10:48.605310 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.605288 2562 scope.go:117] "RemoveContainer" containerID="52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e" Jul 28 15:10:48.614129 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.614109 2562 scope.go:117] "RemoveContainer" containerID="9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5" Jul 28 15:10:48.614409 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:10:48.614392 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5\": container with ID starting with 9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5 not found: ID does not exist" containerID="9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5" Jul 28 15:10:48.614471 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.614414 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5"} err="failed to get container status \"9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5\": rpc error: code = NotFound desc = could not find container \"9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5\": container with ID starting with 9ddda09e0108dcf1d872263a7b2cd9a93d8c21d92c775bab717c3795f29e8ca5 not found: ID does not exist" Jul 28 15:10:48.614471 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.614430 2562 scope.go:117] "RemoveContainer" containerID="52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e" Jul 28 15:10:48.614664 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:10:48.614646 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e\": container with ID starting with 52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e not found: ID does not exist" containerID="52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e" Jul 28 15:10:48.614722 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.614670 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e"} err="failed to get container status \"52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e\": rpc error: code = NotFound desc = could not find container \"52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e\": container with ID starting with 52b2dacac5b73ad42f487df685dea04a398ef268612f1f90665c336beb93846e not found: ID does not exist" Jul 28 15:10:48.622742 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.622721 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc"] Jul 28 15:10:48.626725 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.626703 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-06cd69-openshift-default-575dd9dcb6-thkrc"] Jul 28 15:10:48.888885 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:48.888817 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" path="/var/lib/kubelet/pods/5fb2f071-f672-47b0-977d-6159b501fa99/volumes" Jul 28 15:10:53.560220 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:53.560175 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:53.677506 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:53.677477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:53.778407 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:53.778350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:53.848115 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:53.847977 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6"] Jul 28 15:10:53.850689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:53.850650 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f4853e-7x6s6"] Jul 28 15:10:53.886456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:53.886419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:54.229730 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:54.229624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:10:54.888476 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:54.888440 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b68c2f-b52b-4327-a03d-2ba5d7905244" path="/var/lib/kubelet/pods/45b68c2f-b52b-4327-a03d-2ba5d7905244/volumes" Jul 28 15:10:56.687481 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:10:56.687442 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:11:00.991847 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:00.991817 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:01.083459 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:01.083426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:01.334806 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:01.334769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:01.556689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:01.556635 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:01.936996 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:01.936960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:02.233198 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233116 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj"] Jul 28 15:11:02.233610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233550 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" Jul 28 15:11:02.233610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233564 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" Jul 28 15:11:02.233610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233578 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" Jul 28 15:11:02.233610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233584 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" Jul 28 15:11:02.233610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233591 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b68c2f-b52b-4327-a03d-2ba5d7905244" containerName="revoke-keys" Jul 28 15:11:02.233610 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233597 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b68c2f-b52b-4327-a03d-2ba5d7905244" containerName="revoke-keys" Jul 28 15:11:02.233827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233667 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b68c2f-b52b-4327-a03d-2ba5d7905244" containerName="revoke-keys" Jul 28 15:11:02.233827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233698 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" Jul 28 15:11:02.233827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.233708 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb2f071-f672-47b0-977d-6159b501fa99" containerName="istio-proxy" Jul 28 15:11:02.236813 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.236795 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.239525 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.239503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 15:11:02.239640 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.239527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cef5a5-openshift-default-dockercfg-pxbh2\"" Jul 28 15:11:02.247161 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.247131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj"] Jul 28 15:11:02.255770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.255667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:02.291563 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-socket\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291706 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlpv\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-kube-api-access-lzlpv\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291706 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-certs\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291706 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-envoy\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291706 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-istio-token\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291706 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-data\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291960 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-credential-socket\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291960 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41610-5dff-497d-b9b6-57091054d03a-istiod-ca-cert\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.291960 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.291890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb41610-5dff-497d-b9b6-57091054d03a-istio-podinfo\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392556 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-socket\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392646 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlpv\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-kube-api-access-lzlpv\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392646 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-certs\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392646 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-envoy\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-istio-token\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-data\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.392856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-credential-socket\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393007 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41610-5dff-497d-b9b6-57091054d03a-istiod-ca-cert\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393007 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.392926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb41610-5dff-497d-b9b6-57091054d03a-istio-podinfo\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393110 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.393013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-certs\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393110 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.393063 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-socket\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393233 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.393211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-data\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393296 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.393231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-credential-socket\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.393601 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.393577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41610-5dff-497d-b9b6-57091054d03a-istiod-ca-cert\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.394906 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.394885 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-envoy\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.395352 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.395335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb41610-5dff-497d-b9b6-57091054d03a-istio-podinfo\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.400826 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.400804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-istio-token\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.401032 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.401011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlpv\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-kube-api-access-lzlpv\") pod \"e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.494176 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.494113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:02.547605 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.547584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:02.682314 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.682285 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj"] Jul 28 15:11:02.683582 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:11:02.683558 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb41610_5dff_497d_b9b6_57091054d03a.slice/crio-164defb8bacebb19cb79deb0060a148382386142eefdedde14b4970eba23c696 WatchSource:0}: Error finding container 164defb8bacebb19cb79deb0060a148382386142eefdedde14b4970eba23c696: Status 404 returned error can't find the container with id 164defb8bacebb19cb79deb0060a148382386142eefdedde14b4970eba23c696 Jul 28 15:11:02.685908 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.685877 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:11:02.685985 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.685935 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:11:02.685985 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:02.685964 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:11:03.691850 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:03.691815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" event={"ID":"7cb41610-5dff-497d-b9b6-57091054d03a","Type":"ContainerStarted","Data":"051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6"} Jul 28 15:11:03.692266 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:03.691874 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" event={"ID":"7cb41610-5dff-497d-b9b6-57091054d03a","Type":"ContainerStarted","Data":"164defb8bacebb19cb79deb0060a148382386142eefdedde14b4970eba23c696"} Jul 28 15:11:04.548754 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:04.548712 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:04.553650 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:04.553621 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:04.577391 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:04.577334 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" podStartSLOduration=2.5773168330000003 podStartE2EDuration="2.577316833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:11:03.71602605 +0000 UTC m=+3165.381267872" watchObservedRunningTime="2026-07-28 15:11:04.577316833 +0000 UTC m=+3166.242558657" Jul 28 15:11:04.697671 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:04.697638 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:04.698575 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:04.698558 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:05.790089 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:05.790048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:05.939648 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:05.939612 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:06.038232 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:06.038191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:06.136928 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:06.136895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:06.786014 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:06.785985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:06.924198 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:06.924162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:09.995107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:09.995072 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:11:18.779139 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:18.779108 2562 generic.go:356] "Generic (PLEG): container finished" podID="7cb41610-5dff-497d-b9b6-57091054d03a" containerID="051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6" exitCode=0 Jul 28 15:11:18.779669 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:18.779180 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" event={"ID":"7cb41610-5dff-497d-b9b6-57091054d03a","Type":"ContainerDied","Data":"051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6"} Jul 28 15:11:18.779669 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:18.779608 2562 scope.go:117] "RemoveContainer" containerID="051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6" Jul 28 15:11:18.782043 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:18.780977 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:11:18.782043 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:18.781140 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:11:18.782043 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:18.781236 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:11:19.698058 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:19.698020 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:19.786629 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:19.786595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" event={"ID":"7cb41610-5dff-497d-b9b6-57091054d03a","Type":"ContainerStarted","Data":"7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a"} Jul 28 15:11:20.548139 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:20.548106 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:20.549079 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:20.549056 2562 patch_prober.go:28] interesting pod/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 28 15:11:20.549209 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:20.549100 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 28 15:11:21.548591 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:21.548557 2562 patch_prober.go:28] interesting pod/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 28 15:11:21.549002 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:21.548606 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 28 15:11:22.548088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:22.548054 2562 patch_prober.go:28] interesting pod/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 28 15:11:22.548257 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:22.548116 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 28 15:11:23.548301 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:23.548270 2562 patch_prober.go:28] interesting pod/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 28 15:11:23.548691 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:23.548324 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 28 15:11:24.552456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:24.552424 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:24.552858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:24.552802 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:24.553576 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:24.553556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:11:30.833088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:30.833044 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:31.341126 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:31.341095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:31.729776 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:31.729688 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:32.135340 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:32.135305 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:34.745121 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:34.745091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:34.840277 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:34.840252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:35.098916 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:35.098877 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:35.349918 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:35.349838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:35.484973 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:35.484941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:35.688017 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:35.687921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:35.896705 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:35.896647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:11:43.600066 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:43.600027 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:11:59.884595 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:11:59.884541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:03.059220 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.059185 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx"] Jul 28 15:12:03.062521 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.062501 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:03.065964 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.065933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:12:03.066074 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.066011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 15:12:03.078288 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.078266 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx"] Jul 28 15:12:03.098839 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.098815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b29caf-3a93-46ea-8e11-11f90bafaad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx\" (UID: \"49b29caf-3a93-46ea-8e11-11f90bafaad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:03.199929 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.199899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b29caf-3a93-46ea-8e11-11f90bafaad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx\" (UID: \"49b29caf-3a93-46ea-8e11-11f90bafaad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:03.200469 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.200448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b29caf-3a93-46ea-8e11-11f90bafaad5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx\" (UID: \"49b29caf-3a93-46ea-8e11-11f90bafaad5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:03.373469 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.373410 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:03.506146 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:03.506116 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx"] Jul 28 15:12:03.506762 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:12:03.506735 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b29caf_3a93_46ea_8e11_11f90bafaad5.slice/crio-9d26b09fd7646075471453f0b51be32d87111953f6a4149d9832d35910778939 WatchSource:0}: Error finding container 9d26b09fd7646075471453f0b51be32d87111953f6a4149d9832d35910778939: Status 404 returned error can't find the container with id 9d26b09fd7646075471453f0b51be32d87111953f6a4149d9832d35910778939 Jul 28 15:12:04.042787 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:04.042712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" event={"ID":"49b29caf-3a93-46ea-8e11-11f90bafaad5","Type":"ContainerStarted","Data":"492c8701eb6011c09859cf9dbe2157f5d2c1efbc614bfc5a309bf06654783e3a"} Jul 28 15:12:04.042787 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:04.042784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" event={"ID":"49b29caf-3a93-46ea-8e11-11f90bafaad5","Type":"ContainerStarted","Data":"9d26b09fd7646075471453f0b51be32d87111953f6a4149d9832d35910778939"} Jul 28 15:12:04.061793 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:04.061740 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" podStartSLOduration=1.0617258 podStartE2EDuration="1.0617258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:12:04.059997911 +0000 UTC m=+3225.725239735" watchObservedRunningTime="2026-07-28 15:12:04.0617258 +0000 UTC m=+3225.726967623" Jul 28 15:12:05.050223 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:05.050188 2562 generic.go:356] "Generic (PLEG): container finished" podID="49b29caf-3a93-46ea-8e11-11f90bafaad5" containerID="492c8701eb6011c09859cf9dbe2157f5d2c1efbc614bfc5a309bf06654783e3a" exitCode=0 Jul 28 15:12:05.050486 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:05.050239 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" event={"ID":"49b29caf-3a93-46ea-8e11-11f90bafaad5","Type":"ContainerDied","Data":"492c8701eb6011c09859cf9dbe2157f5d2c1efbc614bfc5a309bf06654783e3a"} Jul 28 15:12:06.191186 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:06.191161 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:06.223913 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:06.223892 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b29caf-3a93-46ea-8e11-11f90bafaad5-maas-api-service-ca\") pod \"49b29caf-3a93-46ea-8e11-11f90bafaad5\" (UID: \"49b29caf-3a93-46ea-8e11-11f90bafaad5\") " Jul 28 15:12:06.224295 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:06.224271 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b29caf-3a93-46ea-8e11-11f90bafaad5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49b29caf-3a93-46ea-8e11-11f90bafaad5" (UID: "49b29caf-3a93-46ea-8e11-11f90bafaad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:12:06.324635 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:06.324574 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b29caf-3a93-46ea-8e11-11f90bafaad5-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:07.063062 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:07.063034 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" Jul 28 15:12:07.063201 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:07.063068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx" event={"ID":"49b29caf-3a93-46ea-8e11-11f90bafaad5","Type":"ContainerDied","Data":"9d26b09fd7646075471453f0b51be32d87111953f6a4149d9832d35910778939"} Jul 28 15:12:07.063201 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:07.063124 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d26b09fd7646075471453f0b51be32d87111953f6a4149d9832d35910778939" Jul 28 15:12:18.653930 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:18.653829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:18.747191 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:18.747161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:18.834473 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:18.834440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:18.880214 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:18.880184 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx"] Jul 28 15:12:18.882411 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:18.882395 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cef5a5-mwfqx"] Jul 28 15:12:18.888922 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:18.888896 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b29caf-3a93-46ea-8e11-11f90bafaad5" path="/var/lib/kubelet/pods/49b29caf-3a93-46ea-8e11-11f90bafaad5/volumes" Jul 28 15:12:19.225518 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:19.225485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:19.331896 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:19.331843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:25.003458 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:25.003424 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:12:25.671219 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:25.671185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:25.775631 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:25.775599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:25.944830 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:25.944739 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj"] Jul 28 15:12:25.945022 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:25.944997 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" containerID="cri-o://7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a" gracePeriod=30 Jul 28 15:12:26.025977 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.025941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:26.233741 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.233646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:26.349573 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.349538 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4"] Jul 28 15:12:26.349916 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.349903 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b29caf-3a93-46ea-8e11-11f90bafaad5" containerName="revoke-keys" Jul 28 15:12:26.349970 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.349917 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b29caf-3a93-46ea-8e11-11f90bafaad5" containerName="revoke-keys" Jul 28 15:12:26.350004 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.349984 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b29caf-3a93-46ea-8e11-11f90bafaad5" containerName="revoke-keys" Jul 28 15:12:26.354206 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.354191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:26.356351 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.356324 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 15:12:26.356351 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.356337 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:12:26.361585 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.361558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4"] Jul 28 15:12:26.389689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.389637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f940d-96d0-4730-9d3f-f67be1d48d42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-becc22-q2wj4\" (UID: \"058f940d-96d0-4730-9d3f-f67be1d48d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:26.490648 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.490563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f940d-96d0-4730-9d3f-f67be1d48d42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-becc22-q2wj4\" (UID: \"058f940d-96d0-4730-9d3f-f67be1d48d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:26.491196 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.491176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f940d-96d0-4730-9d3f-f67be1d48d42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-becc22-q2wj4\" (UID: \"058f940d-96d0-4730-9d3f-f67be1d48d42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:26.666394 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.666356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:26.802724 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:26.802689 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4"] Jul 28 15:12:26.804490 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:12:26.804467 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058f940d_96d0_4730_9d3f_f67be1d48d42.slice/crio-ebd58e3a80bd1590eba6f8813b62b6fe7e40d094c0748a1058e59dcd08b7c9a8 WatchSource:0}: Error finding container ebd58e3a80bd1590eba6f8813b62b6fe7e40d094c0748a1058e59dcd08b7c9a8: Status 404 returned error can't find the container with id ebd58e3a80bd1590eba6f8813b62b6fe7e40d094c0748a1058e59dcd08b7c9a8 Jul 28 15:12:27.180023 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:27.179989 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" event={"ID":"058f940d-96d0-4730-9d3f-f67be1d48d42","Type":"ContainerStarted","Data":"c55ce5a95afbbf0e10583a56af5b0bf0cadeb03b387962870e1af2407328781a"} Jul 28 15:12:27.180413 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:27.180046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" event={"ID":"058f940d-96d0-4730-9d3f-f67be1d48d42","Type":"ContainerStarted","Data":"ebd58e3a80bd1590eba6f8813b62b6fe7e40d094c0748a1058e59dcd08b7c9a8"} Jul 28 15:12:27.206538 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:27.206493 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" podStartSLOduration=1.206476767 podStartE2EDuration="1.206476767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:12:27.204985336 +0000 UTC m=+3248.870227160" watchObservedRunningTime="2026-07-28 15:12:27.206476767 +0000 UTC m=+3248.871718590" Jul 28 15:12:28.187352 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:28.187321 2562 generic.go:356] "Generic (PLEG): container finished" podID="058f940d-96d0-4730-9d3f-f67be1d48d42" containerID="c55ce5a95afbbf0e10583a56af5b0bf0cadeb03b387962870e1af2407328781a" exitCode=0 Jul 28 15:12:28.187739 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:28.187368 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" event={"ID":"058f940d-96d0-4730-9d3f-f67be1d48d42","Type":"ContainerDied","Data":"c55ce5a95afbbf0e10583a56af5b0bf0cadeb03b387962870e1af2407328781a"} Jul 28 15:12:29.326300 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:29.326279 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:29.414726 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:29.414704 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f940d-96d0-4730-9d3f-f67be1d48d42-maas-api-service-ca\") pod \"058f940d-96d0-4730-9d3f-f67be1d48d42\" (UID: \"058f940d-96d0-4730-9d3f-f67be1d48d42\") " Jul 28 15:12:29.415024 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:29.415007 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058f940d-96d0-4730-9d3f-f67be1d48d42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "058f940d-96d0-4730-9d3f-f67be1d48d42" (UID: "058f940d-96d0-4730-9d3f-f67be1d48d42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:12:29.515491 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:29.515435 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058f940d-96d0-4730-9d3f-f67be1d48d42-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:30.200466 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:30.200439 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" Jul 28 15:12:30.200642 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:30.200462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4" event={"ID":"058f940d-96d0-4730-9d3f-f67be1d48d42","Type":"ContainerDied","Data":"ebd58e3a80bd1590eba6f8813b62b6fe7e40d094c0748a1058e59dcd08b7c9a8"} Jul 28 15:12:30.200642 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:30.200507 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd58e3a80bd1590eba6f8813b62b6fe7e40d094c0748a1058e59dcd08b7c9a8" Jul 28 15:12:31.199229 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.199208 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:12:31.207230 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.207174 2562 generic.go:356] "Generic (PLEG): container finished" podID="7cb41610-5dff-497d-b9b6-57091054d03a" containerID="7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a" exitCode=0 Jul 28 15:12:31.207230 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.207210 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" Jul 28 15:12:31.207360 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.207226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" event={"ID":"7cb41610-5dff-497d-b9b6-57091054d03a","Type":"ContainerDied","Data":"7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a"} Jul 28 15:12:31.207360 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.207276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj" event={"ID":"7cb41610-5dff-497d-b9b6-57091054d03a","Type":"ContainerDied","Data":"164defb8bacebb19cb79deb0060a148382386142eefdedde14b4970eba23c696"} Jul 28 15:12:31.207360 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.207300 2562 scope.go:117] "RemoveContainer" containerID="7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a" Jul 28 15:12:31.215776 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.215762 2562 scope.go:117] "RemoveContainer" containerID="051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6" Jul 28 15:12:31.223842 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.223814 2562 scope.go:117] "RemoveContainer" containerID="7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a" Jul 28 15:12:31.224103 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:12:31.224081 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a\": container with ID starting with 7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a not found: ID does not exist" containerID="7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a" Jul 28 15:12:31.224197 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.224107 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a"} err="failed to get container status \"7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a\": rpc error: code = NotFound desc = could not find container \"7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a\": container with ID starting with 7f8c75ee8d5e29b59f35903789a2885acdce30004c0c53d8b67a75135345ed3a not found: ID does not exist" Jul 28 15:12:31.224197 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.224121 2562 scope.go:117] "RemoveContainer" containerID="051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6" Jul 28 15:12:31.224368 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:12:31.224345 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6\": container with ID starting with 051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6 not found: ID does not exist" containerID="051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6" Jul 28 15:12:31.224407 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.224379 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6"} err="failed to get container status \"051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6\": rpc error: code = NotFound desc = could not find container \"051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6\": container with ID starting with 051bbbe35d4e85907b0d128a12553cb5e586a4a77646b47c9d8d021dca1fe1b6 not found: ID does not exist" Jul 28 15:12:31.331206 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-istio-token\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331339 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331218 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-certs\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331339 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331238 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb41610-5dff-497d-b9b6-57091054d03a-istio-podinfo\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331339 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331255 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41610-5dff-497d-b9b6-57091054d03a-istiod-ca-cert\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331339 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331284 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzlpv\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-kube-api-access-lzlpv\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331339 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331303 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-envoy\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331339 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331322 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-credential-socket\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331424 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-socket\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331445 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-data\") pod \"7cb41610-5dff-497d-b9b6-57091054d03a\" (UID: \"7cb41610-5dff-497d-b9b6-57091054d03a\") " Jul 28 15:12:31.331645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331589 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:12:31.331851 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331737 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:12:31.331851 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331756 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.331957 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331903 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb41610-5dff-497d-b9b6-57091054d03a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:12:31.332013 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.331983 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:12:31.332634 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.332608 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:12:31.333582 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.333558 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-kube-api-access-lzlpv" (OuterVolumeSpecName: "kube-api-access-lzlpv") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "kube-api-access-lzlpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:12:31.333582 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.333567 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cb41610-5dff-497d-b9b6-57091054d03a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 15:12:31.333737 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.333586 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:12:31.333737 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.333710 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cb41610-5dff-497d-b9b6-57091054d03a" (UID: "7cb41610-5dff-497d-b9b6-57091054d03a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:12:31.432479 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432453 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432479 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432476 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cb41610-5dff-497d-b9b6-57091054d03a-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432665 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432491 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cb41610-5dff-497d-b9b6-57091054d03a-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432665 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432504 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlpv\" (UniqueName: \"kubernetes.io/projected/7cb41610-5dff-497d-b9b6-57091054d03a-kube-api-access-lzlpv\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432665 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432515 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432665 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432527 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432665 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432540 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.432665 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.432553 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cb41610-5dff-497d-b9b6-57091054d03a-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:12:31.531122 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.531094 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj"] Jul 28 15:12:31.540205 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:31.540187 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cef5a5-openshift-default-5d5c58c7d6-lh6sj"] Jul 28 15:12:32.889873 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:32.889833 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" path="/var/lib/kubelet/pods/7cb41610-5dff-497d-b9b6-57091054d03a/volumes" Jul 28 15:12:35.715870 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:35.715834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:35.887560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:35.887530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:36.054480 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:36.054432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:36.077649 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:36.077620 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4"] Jul 28 15:12:36.080346 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:36.080324 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-becc22-q2wj4"] Jul 28 15:12:36.176573 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:36.176525 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:36.477314 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:36.477200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:36.892196 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:36.892157 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058f940d-96d0-4730-9d3f-f67be1d48d42" path="/var/lib/kubelet/pods/058f940d-96d0-4730-9d3f-f67be1d48d42/volumes" Jul 28 15:12:38.408723 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:38.408683 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:12:43.877857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:43.877813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:44.032051 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:44.031989 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:44.270904 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:44.270823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:44.468869 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:44.468839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:44.800495 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:44.800460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:45.163552 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163480 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6"] Jul 28 15:12:45.163920 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163832 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058f940d-96d0-4730-9d3f-f67be1d48d42" containerName="revoke-keys" Jul 28 15:12:45.163920 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163861 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="058f940d-96d0-4730-9d3f-f67be1d48d42" containerName="revoke-keys" Jul 28 15:12:45.163920 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163880 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" Jul 28 15:12:45.163920 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163885 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" Jul 28 15:12:45.163920 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163895 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" Jul 28 15:12:45.163920 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163901 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" Jul 28 15:12:45.164106 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163952 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="058f940d-96d0-4730-9d3f-f67be1d48d42" containerName="revoke-keys" Jul 28 15:12:45.164106 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163961 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" Jul 28 15:12:45.164106 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.163967 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cb41610-5dff-497d-b9b6-57091054d03a" containerName="istio-proxy" Jul 28 15:12:45.168564 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.168544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.172116 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.172095 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 15:12:45.172231 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.172119 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d96a19-openshift-default-dockercfg-ksth5\"" Jul 28 15:12:45.181399 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.181381 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6"] Jul 28 15:12:45.190227 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.190204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:45.243825 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.243795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaff6d7-097f-442c-ab0b-10aedd31af74-istiod-ca-cert\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.243936 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.243848 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-token\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.243936 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.243902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-envoy\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.244018 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.243969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-podinfo\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.244018 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.243991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-socket\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.244082 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.244032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-data\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.244082 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.244050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-credential-socket\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.244152 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.244087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4cgk\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-kube-api-access-q4cgk\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.244152 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.244106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-certs\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345143 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-podinfo\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345240 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-socket\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345240 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-data\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345240 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-credential-socket\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345376 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cgk\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-kube-api-access-q4cgk\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345376 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-certs\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345513 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaff6d7-097f-442c-ab0b-10aedd31af74-istiod-ca-cert\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345575 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-token\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345632 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-data\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345632 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-envoy\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345635 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-socket\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-credential-socket\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.345770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.345666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-certs\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.346183 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.346161 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaff6d7-097f-442c-ab0b-10aedd31af74-istiod-ca-cert\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.347585 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.347560 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-podinfo\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.347669 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.347641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-envoy\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.360087 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.360066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-token\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.360279 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.360262 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cgk\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-kube-api-access-q4cgk\") pod \"e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.478452 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.478398 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:45.620285 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.620253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:45.656790 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.656765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6"] Jul 28 15:12:45.657507 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:12:45.657483 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaff6d7_097f_442c_ab0b_10aedd31af74.slice/crio-b52423822fe65f67a451d9f98c2c87c2f4e520e8efe1bc9fb6c5110b1b7538ff WatchSource:0}: Error finding container b52423822fe65f67a451d9f98c2c87c2f4e520e8efe1bc9fb6c5110b1b7538ff: Status 404 returned error can't find the container with id b52423822fe65f67a451d9f98c2c87c2f4e520e8efe1bc9fb6c5110b1b7538ff Jul 28 15:12:45.659550 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.659523 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:12:45.659613 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.659583 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:12:45.659650 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:45.659621 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:12:46.302189 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:46.302140 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" event={"ID":"adaff6d7-097f-442c-ab0b-10aedd31af74","Type":"ContainerStarted","Data":"c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43"} Jul 28 15:12:46.302646 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:46.302242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" event={"ID":"adaff6d7-097f-442c-ab0b-10aedd31af74","Type":"ContainerStarted","Data":"b52423822fe65f67a451d9f98c2c87c2f4e520e8efe1bc9fb6c5110b1b7538ff"} Jul 28 15:12:46.329195 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:46.329149 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" podStartSLOduration=1.329134836 podStartE2EDuration="1.329134836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:12:46.325910863 +0000 UTC m=+3267.991152686" watchObservedRunningTime="2026-07-28 15:12:46.329134836 +0000 UTC m=+3267.994376655" Jul 28 15:12:46.479560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:46.479516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:46.484182 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:46.484160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:47.307910 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:47.307877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:47.308915 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:47.308894 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:12:48.491246 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:48.491216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:48.684785 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:48.684753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:48.803844 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:48.803815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:49.051351 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:49.051319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:49.357412 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:49.357376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:49.463837 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:49.463805 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:12:52.614294 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:12:52.614261 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:13:01.387971 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:01.387936 2562 generic.go:356] "Generic (PLEG): container finished" podID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerID="c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43" exitCode=0 Jul 28 15:13:01.388396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:01.388023 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" event={"ID":"adaff6d7-097f-442c-ab0b-10aedd31af74","Type":"ContainerDied","Data":"c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43"} Jul 28 15:13:01.388467 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:01.388432 2562 scope.go:117] "RemoveContainer" containerID="c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43" Jul 28 15:13:01.389608 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:01.389568 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:13:01.389747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:01.389659 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:13:01.389747 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:01.389717 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:13:02.308565 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:02.308515 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:13:02.395992 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:02.395947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" event={"ID":"adaff6d7-097f-442c-ab0b-10aedd31af74","Type":"ContainerStarted","Data":"5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5"} Jul 28 15:13:02.478746 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:02.478711 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:13:02.479762 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:02.479741 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 15:13:02.479861 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:02.479791 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 15:13:03.479090 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:03.479050 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 15:13:03.479484 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:03.479118 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 15:13:04.479134 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:04.479100 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 15:13:04.479515 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:04.479153 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 15:13:05.479745 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:05.479709 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 28 15:13:05.480135 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:05.479766 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 28 15:13:05.983131 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:05.983097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:06.483406 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:06.483377 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:13:06.483814 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:06.483625 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:13:06.484402 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:06.484386 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:13:13.417274 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:13.417240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:13.792975 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:13.792902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:14.235955 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:14.235922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:14.354241 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:14.354208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:17.187287 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:17.187255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:17.324110 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:17.324084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:17.411241 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:17.411202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:17.838029 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:17.837986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:18.040828 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:18.040759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:18.143930 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:18.143846 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:19.134655 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:19.134631 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:13:19.147035 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:19.147010 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:13:24.689910 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:24.689873 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:13:43.319056 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:43.318970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:49.955335 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:49.955303 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:13:59.132692 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:13:59.132637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:14:02.400023 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:14:02.399991 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:14:12.520876 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:14:12.520841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:14:21.992379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:14:21.992343 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:14:25.941507 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:14:25.941469 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:14:48.829808 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:14:48.829775 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:15:00.157432 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.157397 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7"] Jul 28 15:15:00.160858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.160837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" Jul 28 15:15:00.164856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.164835 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:15:00.170598 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.170575 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" Jul 28 15:15:00.172714 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.172694 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd"] Jul 28 15:15:00.176359 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.176338 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z"] Jul 28 15:15:00.176491 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.176469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" Jul 28 15:15:00.180404 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.180381 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7"] Jul 28 15:15:00.180510 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.180475 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" Jul 28 15:15:00.192245 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.192208 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z"] Jul 28 15:15:00.197000 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.196976 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd"] Jul 28 15:15:00.208516 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.208492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" Jul 28 15:15:00.216498 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.216476 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" Jul 28 15:15:00.346258 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.346211 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7"] Jul 28 15:15:00.347650 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:15:00.347623 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6cdb21_4194_42f8_a843_419e96095274.slice/crio-db430ba3eee60d073706932cfd0bbc0443a5f75a455b14e78d9e9aed8ce42f8c WatchSource:0}: Error finding container db430ba3eee60d073706932cfd0bbc0443a5f75a455b14e78d9e9aed8ce42f8c: Status 404 returned error can't find the container with id db430ba3eee60d073706932cfd0bbc0443a5f75a455b14e78d9e9aed8ce42f8c Jul 28 15:15:00.573509 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.573484 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd"] Jul 28 15:15:00.574926 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:15:00.574900 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608eb65f_2368_47e8_b1b9_69efafdc0ba2.slice/crio-64c4854b5803428c81757400a51959e7fc007851ba06bc458acc9083c6cd95e0 WatchSource:0}: Error finding container 64c4854b5803428c81757400a51959e7fc007851ba06bc458acc9083c6cd95e0: Status 404 returned error can't find the container with id 64c4854b5803428c81757400a51959e7fc007851ba06bc458acc9083c6cd95e0 Jul 28 15:15:00.603126 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:00.603102 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z"] Jul 28 15:15:00.605855 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:15:00.605831 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ba50e5_ac74_4c0d_90f3_3f1255456639.slice/crio-a2e58d9692288456e43191b9cd24ecac1eda0ea055ba61a98f937e9d23e14c96 WatchSource:0}: Error finding container a2e58d9692288456e43191b9cd24ecac1eda0ea055ba61a98f937e9d23e14c96: Status 404 returned error can't find the container with id a2e58d9692288456e43191b9cd24ecac1eda0ea055ba61a98f937e9d23e14c96 Jul 28 15:15:01.069840 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.069807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" event={"ID":"608eb65f-2368-47e8-b1b9-69efafdc0ba2","Type":"ContainerStarted","Data":"ecde5c574be225a8cacab9899fa76865d15c2afe79b4b3f6a5d37b3f0c895897"} Jul 28 15:15:01.069992 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.069881 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" event={"ID":"608eb65f-2368-47e8-b1b9-69efafdc0ba2","Type":"ContainerStarted","Data":"64c4854b5803428c81757400a51959e7fc007851ba06bc458acc9083c6cd95e0"} Jul 28 15:15:01.071530 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.071499 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" event={"ID":"a0ba50e5-ac74-4c0d-90f3-3f1255456639","Type":"ContainerStarted","Data":"49bf7869791f04662c4550d74ccfb03974a07d465feb39152e8bed1d91d48fcf"} Jul 28 15:15:01.071835 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.071556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" event={"ID":"a0ba50e5-ac74-4c0d-90f3-3f1255456639","Type":"ContainerStarted","Data":"a2e58d9692288456e43191b9cd24ecac1eda0ea055ba61a98f937e9d23e14c96"} Jul 28 15:15:01.073234 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.073163 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" event={"ID":"7c6cdb21-4194-42f8-a843-419e96095274","Type":"ContainerStarted","Data":"f6161683eda193c92f1f833ebe4114197756ced45ed3831c2f772e551cc6dd01"} Jul 28 15:15:01.073234 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.073207 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" event={"ID":"7c6cdb21-4194-42f8-a843-419e96095274","Type":"ContainerStarted","Data":"db430ba3eee60d073706932cfd0bbc0443a5f75a455b14e78d9e9aed8ce42f8c"} Jul 28 15:15:01.103864 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.103816 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" podStartSLOduration=1.103800509 podStartE2EDuration="1.103800509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:15:01.100327041 +0000 UTC m=+3402.765568863" watchObservedRunningTime="2026-07-28 15:15:01.103800509 +0000 UTC m=+3402.769042335" Jul 28 15:15:01.125511 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.125469 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" podStartSLOduration=1.125457389 podStartE2EDuration="1.125457389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:15:01.121769111 +0000 UTC m=+3402.787010934" watchObservedRunningTime="2026-07-28 15:15:01.125457389 +0000 UTC m=+3402.790699212" Jul 28 15:15:01.150094 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:01.150054 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" podStartSLOduration=1.150042588 podStartE2EDuration="1.150042588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:15:01.148524487 +0000 UTC m=+3402.813766309" watchObservedRunningTime="2026-07-28 15:15:01.150042588 +0000 UTC m=+3402.815284410" Jul 28 15:15:02.080526 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:02.080491 2562 generic.go:356] "Generic (PLEG): container finished" podID="7c6cdb21-4194-42f8-a843-419e96095274" containerID="f6161683eda193c92f1f833ebe4114197756ced45ed3831c2f772e551cc6dd01" exitCode=0 Jul 28 15:15:02.080999 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:02.080579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" event={"ID":"7c6cdb21-4194-42f8-a843-419e96095274","Type":"ContainerDied","Data":"f6161683eda193c92f1f833ebe4114197756ced45ed3831c2f772e551cc6dd01"} Jul 28 15:15:02.082225 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:02.082203 2562 generic.go:356] "Generic (PLEG): container finished" podID="608eb65f-2368-47e8-b1b9-69efafdc0ba2" containerID="ecde5c574be225a8cacab9899fa76865d15c2afe79b4b3f6a5d37b3f0c895897" exitCode=0 Jul 28 15:15:02.082378 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:02.082288 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" event={"ID":"608eb65f-2368-47e8-b1b9-69efafdc0ba2","Type":"ContainerDied","Data":"ecde5c574be225a8cacab9899fa76865d15c2afe79b4b3f6a5d37b3f0c895897"} Jul 28 15:15:02.083709 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:02.083666 2562 generic.go:356] "Generic (PLEG): container finished" podID="a0ba50e5-ac74-4c0d-90f3-3f1255456639" containerID="49bf7869791f04662c4550d74ccfb03974a07d465feb39152e8bed1d91d48fcf" exitCode=0 Jul 28 15:15:02.083709 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:02.083696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" event={"ID":"a0ba50e5-ac74-4c0d-90f3-3f1255456639","Type":"ContainerDied","Data":"49bf7869791f04662c4550d74ccfb03974a07d465feb39152e8bed1d91d48fcf"} Jul 28 15:15:03.252469 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:03.252447 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" Jul 28 15:15:03.279717 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:03.279696 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" Jul 28 15:15:03.283779 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:03.283761 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" Jul 28 15:15:04.097972 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.097944 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" Jul 28 15:15:04.098161 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.097989 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z" event={"ID":"a0ba50e5-ac74-4c0d-90f3-3f1255456639","Type":"ContainerDied","Data":"a2e58d9692288456e43191b9cd24ecac1eda0ea055ba61a98f937e9d23e14c96"} Jul 28 15:15:04.098161 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.098040 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e58d9692288456e43191b9cd24ecac1eda0ea055ba61a98f937e9d23e14c96" Jul 28 15:15:04.099496 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.099477 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" Jul 28 15:15:04.099622 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.099507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7" event={"ID":"7c6cdb21-4194-42f8-a843-419e96095274","Type":"ContainerDied","Data":"db430ba3eee60d073706932cfd0bbc0443a5f75a455b14e78d9e9aed8ce42f8c"} Jul 28 15:15:04.099622 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.099557 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db430ba3eee60d073706932cfd0bbc0443a5f75a455b14e78d9e9aed8ce42f8c" Jul 28 15:15:04.101158 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.101135 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" event={"ID":"608eb65f-2368-47e8-b1b9-69efafdc0ba2","Type":"ContainerDied","Data":"64c4854b5803428c81757400a51959e7fc007851ba06bc458acc9083c6cd95e0"} Jul 28 15:15:04.101280 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.101173 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754195-vm8vd" Jul 28 15:15:04.101280 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.101180 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64c4854b5803428c81757400a51959e7fc007851ba06bc458acc9083c6cd95e0" Jul 28 15:15:04.389706 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.389615 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq"] Jul 28 15:15:04.393074 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.393052 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754150-zdcrq"] Jul 28 15:15:04.889062 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:04.889038 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808c9b69-75e0-4232-a5bb-0b7d244aabad" path="/var/lib/kubelet/pods/808c9b69-75e0-4232-a5bb-0b7d244aabad/volumes" Jul 28 15:15:09.809194 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:09.809159 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:15:19.311341 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:19.311260 2562 scope.go:117] "RemoveContainer" containerID="7e92c183c745eb2be6c1b5ea785b54dc5fdbe670f0d45243f6bc60d2cc4e69b3" Jul 28 15:15:23.312966 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:23.312932 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:15:50.211625 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:15:50.211575 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:16:07.010118 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:16:07.010084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:16:28.429523 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:16:28.429481 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:16:45.992869 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:16:45.992793 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:16:46.247300 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:16:46.247201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:17:06.620437 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:06.620389 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:17:07.003571 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:07.003476 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:17:08.438577 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:08.438536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:17:19.341663 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:19.341631 2562 scope.go:117] "RemoveContainer" containerID="2d187682b88582d2fdd68c755c9bb699ee58e01533e84242f1d783b452286171" Jul 28 15:17:19.352940 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:19.352915 2562 scope.go:117] "RemoveContainer" containerID="a0c2e9d41219adef3b9637d54f4b92724d6870b10c64479e0d80d96ee0911551" Jul 28 15:17:25.717530 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:25.717492 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:17:49.739282 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:17:49.739242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:18:03.022438 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:03.022398 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:18:19.170415 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:19.170386 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:18:19.184727 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:19.184701 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:18:19.376796 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:19.376762 2562 scope.go:117] "RemoveContainer" containerID="492c8701eb6011c09859cf9dbe2157f5d2c1efbc614bfc5a309bf06654783e3a" Jul 28 15:18:24.100599 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:24.100556 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:18:29.657720 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:29.657687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:18:30.568091 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:30.568049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:18:40.027171 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:40.027091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:18:41.314228 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:18:41.314200 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:19:09.046197 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:09.046159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:19:12.394133 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:12.394101 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:19:12.994942 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:12.994900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:19:19.398141 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:19.398106 2562 scope.go:117] "RemoveContainer" containerID="c55ce5a95afbbf0e10583a56af5b0bf0cadeb03b387962870e1af2407328781a" Jul 28 15:19:30.661216 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:30.661177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:19:30.754784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:30.754754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:19:40.291890 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:40.291861 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:19:43.912705 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:43.912655 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:19:48.790267 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:19:48.790232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:20:30.066128 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:20:30.066083 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:20:37.402602 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:20:37.402527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:20:37.531693 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:20:37.531643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:20:40.693501 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:20:40.693461 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:20:59.595059 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:20:59.595017 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:21:19.828005 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:19.827900 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:21:25.162082 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:25.162036 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:21:26.003217 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:26.003183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:21:34.598692 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:34.598621 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:21:37.389950 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:37.389918 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:21:56.540113 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:56.540067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:21:59.898685 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:21:59.898638 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:22:29.606321 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:29.606281 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:22:48.543900 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:48.543818 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:22:57.004350 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:57.004313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:22:59.063103 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063065 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7424-140452070871472"] Jul 28 15:22:59.063478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063397 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6cdb21-4194-42f8-a843-419e96095274" containerName="cleanup" Jul 28 15:22:59.063478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063408 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6cdb21-4194-42f8-a843-419e96095274" containerName="cleanup" Jul 28 15:22:59.063478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063423 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ba50e5-ac74-4c0d-90f3-3f1255456639" containerName="cleanup" Jul 28 15:22:59.063478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063428 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ba50e5-ac74-4c0d-90f3-3f1255456639" containerName="cleanup" Jul 28 15:22:59.063478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063442 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="608eb65f-2368-47e8-b1b9-69efafdc0ba2" containerName="cleanup" Jul 28 15:22:59.063478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063460 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="608eb65f-2368-47e8-b1b9-69efafdc0ba2" containerName="cleanup" Jul 28 15:22:59.063745 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063516 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6cdb21-4194-42f8-a843-419e96095274" containerName="cleanup" Jul 28 15:22:59.063745 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063524 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ba50e5-ac74-4c0d-90f3-3f1255456639" containerName="cleanup" Jul 28 15:22:59.063745 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.063530 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="608eb65f-2368-47e8-b1b9-69efafdc0ba2" containerName="cleanup" Jul 28 15:22:59.066618 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.066600 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:22:59.071164 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.071145 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ctx62\"" Jul 28 15:22:59.097106 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.097084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070871472"] Jul 28 15:22:59.206853 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.206833 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5sf\" (UniqueName: \"kubernetes.io/projected/ef1ad75d-7b80-4b77-a650-42232131868d-kube-api-access-9m5sf\") pod \"test-curl-7424-140452070871472\" (UID: \"ef1ad75d-7b80-4b77-a650-42232131868d\") " pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:22:59.307313 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.307289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5sf\" (UniqueName: \"kubernetes.io/projected/ef1ad75d-7b80-4b77-a650-42232131868d-kube-api-access-9m5sf\") pod \"test-curl-7424-140452070871472\" (UID: \"ef1ad75d-7b80-4b77-a650-42232131868d\") " pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:22:59.324640 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.324586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5sf\" (UniqueName: \"kubernetes.io/projected/ef1ad75d-7b80-4b77-a650-42232131868d-kube-api-access-9m5sf\") pod \"test-curl-7424-140452070871472\" (UID: \"ef1ad75d-7b80-4b77-a650-42232131868d\") " pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:22:59.375263 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.375243 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:22:59.536668 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.536648 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070871472"] Jul 28 15:22:59.538852 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:22:59.538825 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef1ad75d_7b80_4b77_a650_42232131868d.slice/crio-827e93e9df2d85e0087ce32fd63d9436f719f17203376d5993240b6ac6f91d89 WatchSource:0}: Error finding container 827e93e9df2d85e0087ce32fd63d9436f719f17203376d5993240b6ac6f91d89: Status 404 returned error can't find the container with id 827e93e9df2d85e0087ce32fd63d9436f719f17203376d5993240b6ac6f91d89 Jul 28 15:22:59.540072 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.540053 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:22:59.599939 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.599905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:22:59.801683 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.801636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070871472" event={"ID":"ef1ad75d-7b80-4b77-a650-42232131868d","Type":"ContainerStarted","Data":"827e93e9df2d85e0087ce32fd63d9436f719f17203376d5993240b6ac6f91d89"} Jul 28 15:22:59.945586 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:22:59.945498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:00.808837 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:00.808798 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef1ad75d-7b80-4b77-a650-42232131868d" containerID="be2dd47c0ec2d8d64f042d8a10b7d7bcf1837e95de23f2789f7a6719feb13221" exitCode=0 Jul 28 15:23:00.809274 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:00.808866 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070871472" event={"ID":"ef1ad75d-7b80-4b77-a650-42232131868d","Type":"ContainerDied","Data":"be2dd47c0ec2d8d64f042d8a10b7d7bcf1837e95de23f2789f7a6719feb13221"} Jul 28 15:23:01.952201 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:01.952176 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:23:02.012835 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.012813 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7424-140452070871472_ef1ad75d-7b80-4b77-a650-42232131868d/test-curl-7424-140452070871472/0.log" Jul 28 15:23:02.031630 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.031606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m5sf\" (UniqueName: \"kubernetes.io/projected/ef1ad75d-7b80-4b77-a650-42232131868d-kube-api-access-9m5sf\") pod \"ef1ad75d-7b80-4b77-a650-42232131868d\" (UID: \"ef1ad75d-7b80-4b77-a650-42232131868d\") " Jul 28 15:23:02.033608 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.033587 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1ad75d-7b80-4b77-a650-42232131868d-kube-api-access-9m5sf" (OuterVolumeSpecName: "kube-api-access-9m5sf") pod "ef1ad75d-7b80-4b77-a650-42232131868d" (UID: "ef1ad75d-7b80-4b77-a650-42232131868d"). InnerVolumeSpecName "kube-api-access-9m5sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:23:02.078444 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.078386 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7424-140452070871472"] Jul 28 15:23:02.082721 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.082702 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7424-140452070871472"] Jul 28 15:23:02.132411 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.132386 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5sf\" (UniqueName: \"kubernetes.io/projected/ef1ad75d-7b80-4b77-a650-42232131868d-kube-api-access-9m5sf\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:23:02.238139 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.238117 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7424-140452070873264"] Jul 28 15:23:02.238470 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.238456 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef1ad75d-7b80-4b77-a650-42232131868d" containerName="test-curl-7424-140452070871472" Jul 28 15:23:02.238526 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.238471 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef1ad75d-7b80-4b77-a650-42232131868d" containerName="test-curl-7424-140452070871472" Jul 28 15:23:02.238602 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.238589 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef1ad75d-7b80-4b77-a650-42232131868d" containerName="test-curl-7424-140452070871472" Jul 28 15:23:02.241565 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.241551 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:02.274635 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.274615 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070873264"] Jul 28 15:23:02.333941 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.333887 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85r9p\" (UniqueName: \"kubernetes.io/projected/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc-kube-api-access-85r9p\") pod \"test-curl-7424-140452070873264\" (UID: \"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc\") " pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:02.434267 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.434240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85r9p\" (UniqueName: \"kubernetes.io/projected/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc-kube-api-access-85r9p\") pod \"test-curl-7424-140452070873264\" (UID: \"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc\") " pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:02.445304 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.445281 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85r9p\" (UniqueName: \"kubernetes.io/projected/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc-kube-api-access-85r9p\") pod \"test-curl-7424-140452070873264\" (UID: \"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc\") " pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:02.550735 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.550716 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:02.703493 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.703462 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070873264"] Jul 28 15:23:02.711914 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:23:02.711885 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd02d9bdd_0ec0_4829_afa4_1f00f526c8cc.slice/crio-4ca01ace36cbe0083cb2a040f9e35ef3ed2b9faf3710f71ad9074f605b45009f WatchSource:0}: Error finding container 4ca01ace36cbe0083cb2a040f9e35ef3ed2b9faf3710f71ad9074f605b45009f: Status 404 returned error can't find the container with id 4ca01ace36cbe0083cb2a040f9e35ef3ed2b9faf3710f71ad9074f605b45009f Jul 28 15:23:02.820546 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.820523 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:23:02.820645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.820542 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="827e93e9df2d85e0087ce32fd63d9436f719f17203376d5993240b6ac6f91d89" Jul 28 15:23:02.821736 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.821712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070873264" event={"ID":"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc","Type":"ContainerStarted","Data":"4ca01ace36cbe0083cb2a040f9e35ef3ed2b9faf3710f71ad9074f605b45009f"} Jul 28 15:23:02.831316 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:23:02.831292 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7424-140452070871472\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="ef1ad75d-7b80-4b77-a650-42232131868d" pod="opendatahub/test-curl-7424-140452070871472" Jul 28 15:23:02.888051 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:02.887999 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef1ad75d-7b80-4b77-a650-42232131868d" path="/var/lib/kubelet/pods/ef1ad75d-7b80-4b77-a650-42232131868d/volumes" Jul 28 15:23:03.340569 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:03.340542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:23:03.830256 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:03.830221 2562 generic.go:356] "Generic (PLEG): container finished" podID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" containerID="390697b8211a7769a8e4bf05ab4fcefab86ea88aea7903410e296b494b25bb89" exitCode=0 Jul 28 15:23:03.830421 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:03.830317 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070873264" event={"ID":"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc","Type":"ContainerDied","Data":"390697b8211a7769a8e4bf05ab4fcefab86ea88aea7903410e296b494b25bb89"} Jul 28 15:23:04.977777 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:04.977753 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:05.026787 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.026761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7424-140452070873264_d02d9bdd-0ec0-4829-afa4-1f00f526c8cc/test-curl-7424-140452070873264/0.log" Jul 28 15:23:05.051831 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.051806 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85r9p\" (UniqueName: \"kubernetes.io/projected/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc-kube-api-access-85r9p\") pod \"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc\" (UID: \"d02d9bdd-0ec0-4829-afa4-1f00f526c8cc\") " Jul 28 15:23:05.053658 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.053636 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc-kube-api-access-85r9p" (OuterVolumeSpecName: "kube-api-access-85r9p") pod "d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" (UID: "d02d9bdd-0ec0-4829-afa4-1f00f526c8cc"). InnerVolumeSpecName "kube-api-access-85r9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:23:05.083667 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.083644 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7424-140452070873264"] Jul 28 15:23:05.093353 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.093297 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7424-140452070873264"] Jul 28 15:23:05.152373 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.152349 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85r9p\" (UniqueName: \"kubernetes.io/projected/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc-kube-api-access-85r9p\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:23:05.283393 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.283372 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:05.283716 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.283704 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" containerName="test-curl-7424-140452070873264" Jul 28 15:23:05.283764 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.283718 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" containerName="test-curl-7424-140452070873264" Jul 28 15:23:05.283800 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.283791 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" containerName="test-curl-7424-140452070873264" Jul 28 15:23:05.286643 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.286628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:05.300836 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.300811 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:05.454121 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.454061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bx5\" (UniqueName: \"kubernetes.io/projected/ebffa749-7ba8-4113-bf3b-11389b1ece49-kube-api-access-z6bx5\") pod \"test-curl-7424-140452070996416\" (UID: \"ebffa749-7ba8-4113-bf3b-11389b1ece49\") " pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:05.554961 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.554935 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bx5\" (UniqueName: \"kubernetes.io/projected/ebffa749-7ba8-4113-bf3b-11389b1ece49-kube-api-access-z6bx5\") pod \"test-curl-7424-140452070996416\" (UID: \"ebffa749-7ba8-4113-bf3b-11389b1ece49\") " pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:05.569146 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.569126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bx5\" (UniqueName: \"kubernetes.io/projected/ebffa749-7ba8-4113-bf3b-11389b1ece49-kube-api-access-z6bx5\") pod \"test-curl-7424-140452070996416\" (UID: \"ebffa749-7ba8-4113-bf3b-11389b1ece49\") " pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:05.595237 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.595216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:05.741588 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.741564 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:05.743333 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:23:05.743307 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebffa749_7ba8_4113_bf3b_11389b1ece49.slice/crio-0ff2baa245542c36629eb49ad0f9b483d3a41eaa01151f9157b30173c533683d WatchSource:0}: Error finding container 0ff2baa245542c36629eb49ad0f9b483d3a41eaa01151f9157b30173c533683d: Status 404 returned error can't find the container with id 0ff2baa245542c36629eb49ad0f9b483d3a41eaa01151f9157b30173c533683d Jul 28 15:23:05.842569 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.842551 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:05.842703 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.842550 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ca01ace36cbe0083cb2a040f9e35ef3ed2b9faf3710f71ad9074f605b45009f" Jul 28 15:23:05.843805 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:05.843776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070996416" event={"ID":"ebffa749-7ba8-4113-bf3b-11389b1ece49","Type":"ContainerStarted","Data":"0ff2baa245542c36629eb49ad0f9b483d3a41eaa01151f9157b30173c533683d"} Jul 28 15:23:05.852839 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:23:05.852813 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7424-140452070873264\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:06.851056 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:06.851025 2562 generic.go:356] "Generic (PLEG): container finished" podID="ebffa749-7ba8-4113-bf3b-11389b1ece49" containerID="57cbaa23fb0609ff6b58106df9eb1e25f183ac45fe7b06100ad919b8f0470124" exitCode=0 Jul 28 15:23:06.851425 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:06.851073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070996416" event={"ID":"ebffa749-7ba8-4113-bf3b-11389b1ece49","Type":"ContainerDied","Data":"57cbaa23fb0609ff6b58106df9eb1e25f183ac45fe7b06100ad919b8f0470124"} Jul 28 15:23:06.888805 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:06.888758 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" path="/var/lib/kubelet/pods/d02d9bdd-0ec0-4829-afa4-1f00f526c8cc/volumes" Jul 28 15:23:06.892839 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:23:06.892811 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7424-140452070873264\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="d02d9bdd-0ec0-4829-afa4-1f00f526c8cc" pod="opendatahub/test-curl-7424-140452070873264" Jul 28 15:23:07.989510 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:07.989488 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.030305 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.030281 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7424-140452070996416_ebffa749-7ba8-4113-bf3b-11389b1ece49/test-curl-7424-140452070996416/0.log" Jul 28 15:23:08.078330 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.078302 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bx5\" (UniqueName: \"kubernetes.io/projected/ebffa749-7ba8-4113-bf3b-11389b1ece49-kube-api-access-z6bx5\") pod \"ebffa749-7ba8-4113-bf3b-11389b1ece49\" (UID: \"ebffa749-7ba8-4113-bf3b-11389b1ece49\") " Jul 28 15:23:08.080190 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.080165 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebffa749-7ba8-4113-bf3b-11389b1ece49-kube-api-access-z6bx5" (OuterVolumeSpecName: "kube-api-access-z6bx5") pod "ebffa749-7ba8-4113-bf3b-11389b1ece49" (UID: "ebffa749-7ba8-4113-bf3b-11389b1ece49"). InnerVolumeSpecName "kube-api-access-z6bx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:23:08.086161 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.085872 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:08.105775 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.105751 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:08.179989 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.179903 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bx5\" (UniqueName: \"kubernetes.io/projected/ebffa749-7ba8-4113-bf3b-11389b1ece49-kube-api-access-z6bx5\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:23:08.284650 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.284613 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:08.284994 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.284980 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebffa749-7ba8-4113-bf3b-11389b1ece49" containerName="test-curl-7424-140452070996416" Jul 28 15:23:08.285039 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.284997 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebffa749-7ba8-4113-bf3b-11389b1ece49" containerName="test-curl-7424-140452070996416" Jul 28 15:23:08.285071 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.285061 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebffa749-7ba8-4113-bf3b-11389b1ece49" containerName="test-curl-7424-140452070996416" Jul 28 15:23:08.287973 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.287955 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.305318 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.305282 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:08.381356 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.381328 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psl7\" (UniqueName: \"kubernetes.io/projected/a842d6fc-45d2-41a1-b223-a2a676ec19d4-kube-api-access-7psl7\") pod \"test-curl-7424-140452070996416\" (UID: \"a842d6fc-45d2-41a1-b223-a2a676ec19d4\") " pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.482487 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.482394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psl7\" (UniqueName: \"kubernetes.io/projected/a842d6fc-45d2-41a1-b223-a2a676ec19d4-kube-api-access-7psl7\") pod \"test-curl-7424-140452070996416\" (UID: \"a842d6fc-45d2-41a1-b223-a2a676ec19d4\") " pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.492235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.492212 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psl7\" (UniqueName: \"kubernetes.io/projected/a842d6fc-45d2-41a1-b223-a2a676ec19d4-kube-api-access-7psl7\") pod \"test-curl-7424-140452070996416\" (UID: \"a842d6fc-45d2-41a1-b223-a2a676ec19d4\") " pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.596783 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.596751 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.759174 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.759151 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:08.761036 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:23:08.761000 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda842d6fc_45d2_41a1_b223_a2a676ec19d4.slice/crio-50029254f7ee7864339152e4c82d7587c5ccc309529b5909dc6146669ab2bccd WatchSource:0}: Error finding container 50029254f7ee7864339152e4c82d7587c5ccc309529b5909dc6146669ab2bccd: Status 404 returned error can't find the container with id 50029254f7ee7864339152e4c82d7587c5ccc309529b5909dc6146669ab2bccd Jul 28 15:23:08.862311 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.862272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070996416" event={"ID":"a842d6fc-45d2-41a1-b223-a2a676ec19d4","Type":"ContainerStarted","Data":"50029254f7ee7864339152e4c82d7587c5ccc309529b5909dc6146669ab2bccd"} Jul 28 15:23:08.863403 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.863388 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:08.863496 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.863412 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff2baa245542c36629eb49ad0f9b483d3a41eaa01151f9157b30173c533683d" Jul 28 15:23:08.876740 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.876710 2562 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7424-140452070996416" oldPodUID="ebffa749-7ba8-4113-bf3b-11389b1ece49" podUID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" Jul 28 15:23:08.893478 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:08.893431 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebffa749-7ba8-4113-bf3b-11389b1ece49" path="/var/lib/kubelet/pods/ebffa749-7ba8-4113-bf3b-11389b1ece49/volumes" Jul 28 15:23:09.870511 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:09.870475 2562 generic.go:356] "Generic (PLEG): container finished" podID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" containerID="4fd8e6b4c1f9f02113be1d631c51282ae83afb199b38b0c89ff485d8dbf20f8b" exitCode=0 Jul 28 15:23:09.870939 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:09.870563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070996416" event={"ID":"a842d6fc-45d2-41a1-b223-a2a676ec19d4","Type":"ContainerDied","Data":"4fd8e6b4c1f9f02113be1d631c51282ae83afb199b38b0c89ff485d8dbf20f8b"} Jul 28 15:23:11.021769 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.021748 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:11.062657 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.062632 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7424-140452070996416_a842d6fc-45d2-41a1-b223-a2a676ec19d4/test-curl-7424-140452070996416/0.log" Jul 28 15:23:11.101819 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.101784 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:11.105573 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.105550 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7424-140452070996416"] Jul 28 15:23:11.108347 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.108315 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7psl7\" (UniqueName: \"kubernetes.io/projected/a842d6fc-45d2-41a1-b223-a2a676ec19d4-kube-api-access-7psl7\") pod \"a842d6fc-45d2-41a1-b223-a2a676ec19d4\" (UID: \"a842d6fc-45d2-41a1-b223-a2a676ec19d4\") " Jul 28 15:23:11.110356 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.110337 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a842d6fc-45d2-41a1-b223-a2a676ec19d4-kube-api-access-7psl7" (OuterVolumeSpecName: "kube-api-access-7psl7") pod "a842d6fc-45d2-41a1-b223-a2a676ec19d4" (UID: "a842d6fc-45d2-41a1-b223-a2a676ec19d4"). InnerVolumeSpecName "kube-api-access-7psl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:23:11.209158 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.209071 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7psl7\" (UniqueName: \"kubernetes.io/projected/a842d6fc-45d2-41a1-b223-a2a676ec19d4-kube-api-access-7psl7\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:23:11.274977 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.274937 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7424-140452070997568"] Jul 28 15:23:11.275348 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.275330 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" containerName="test-curl-7424-140452070996416" Jul 28 15:23:11.275437 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.275351 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" containerName="test-curl-7424-140452070996416" Jul 28 15:23:11.275492 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.275473 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" containerName="test-curl-7424-140452070996416" Jul 28 15:23:11.278567 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.278548 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:11.289900 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.289878 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070997568"] Jul 28 15:23:11.410194 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.410155 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfx9\" (UniqueName: \"kubernetes.io/projected/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c-kube-api-access-8rfx9\") pod \"test-curl-7424-140452070997568\" (UID: \"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c\") " pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:11.511489 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.511394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfx9\" (UniqueName: \"kubernetes.io/projected/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c-kube-api-access-8rfx9\") pod \"test-curl-7424-140452070997568\" (UID: \"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c\") " pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:11.523708 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.523662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfx9\" (UniqueName: \"kubernetes.io/projected/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c-kube-api-access-8rfx9\") pod \"test-curl-7424-140452070997568\" (UID: \"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c\") " pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:11.587598 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.587570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:11.748860 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.748830 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-140452070997568"] Jul 28 15:23:11.751027 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:23:11.751001 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7fa1a59_2e35_4345_8ab0_e4b18a918a4c.slice/crio-46f95f8d7abfb0fbf5bec9d1641367fdd2f461e3c4c42cea2df5f6f4df4f96ce WatchSource:0}: Error finding container 46f95f8d7abfb0fbf5bec9d1641367fdd2f461e3c4c42cea2df5f6f4df4f96ce: Status 404 returned error can't find the container with id 46f95f8d7abfb0fbf5bec9d1641367fdd2f461e3c4c42cea2df5f6f4df4f96ce Jul 28 15:23:11.883890 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.883858 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070997568" event={"ID":"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c","Type":"ContainerStarted","Data":"46f95f8d7abfb0fbf5bec9d1641367fdd2f461e3c4c42cea2df5f6f4df4f96ce"} Jul 28 15:23:11.885075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.885057 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:11.885075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:11.885066 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50029254f7ee7864339152e4c82d7587c5ccc309529b5909dc6146669ab2bccd" Jul 28 15:23:11.899711 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:23:11.899668 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7424-140452070996416\" is forbidden: User \"system:node:ip-10-0-131-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-121.ec2.internal' and this object" podUID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" pod="opendatahub/test-curl-7424-140452070996416" Jul 28 15:23:12.889004 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:12.888975 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a842d6fc-45d2-41a1-b223-a2a676ec19d4" path="/var/lib/kubelet/pods/a842d6fc-45d2-41a1-b223-a2a676ec19d4/volumes" Jul 28 15:23:12.892116 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:12.892087 2562 generic.go:356] "Generic (PLEG): container finished" podID="b7fa1a59-2e35-4345-8ab0-e4b18a918a4c" containerID="fc3a96e4fba0d7e0587d567e68514757d6680d7b9cc9e621b8066715cef19fb2" exitCode=0 Jul 28 15:23:12.892265 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:12.892173 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-140452070997568" event={"ID":"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c","Type":"ContainerDied","Data":"fc3a96e4fba0d7e0587d567e68514757d6680d7b9cc9e621b8066715cef19fb2"} Jul 28 15:23:14.036524 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.036497 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:14.061116 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.061082 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7424-140452070997568_b7fa1a59-2e35-4345-8ab0-e4b18a918a4c/test-curl-7424-140452070997568/0.log" Jul 28 15:23:14.091524 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.091488 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7424-140452070997568"] Jul 28 15:23:14.098165 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.098133 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7424-140452070997568"] Jul 28 15:23:14.132883 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.132852 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rfx9\" (UniqueName: \"kubernetes.io/projected/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c-kube-api-access-8rfx9\") pod \"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c\" (UID: \"b7fa1a59-2e35-4345-8ab0-e4b18a918a4c\") " Jul 28 15:23:14.135015 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.134991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c-kube-api-access-8rfx9" (OuterVolumeSpecName: "kube-api-access-8rfx9") pod "b7fa1a59-2e35-4345-8ab0-e4b18a918a4c" (UID: "b7fa1a59-2e35-4345-8ab0-e4b18a918a4c"). InnerVolumeSpecName "kube-api-access-8rfx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:23:14.233980 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.233883 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfx9\" (UniqueName: \"kubernetes.io/projected/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c-kube-api-access-8rfx9\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:23:14.888898 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.888866 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7fa1a59-2e35-4345-8ab0-e4b18a918a4c" path="/var/lib/kubelet/pods/b7fa1a59-2e35-4345-8ab0-e4b18a918a4c/volumes" Jul 28 15:23:14.903617 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.903594 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-140452070997568" Jul 28 15:23:14.903767 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:14.903655 2562 scope.go:117] "RemoveContainer" containerID="fc3a96e4fba0d7e0587d567e68514757d6680d7b9cc9e621b8066715cef19fb2" Jul 28 15:23:17.600805 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:17.600770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:18.181158 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.181125 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5"] Jul 28 15:23:18.181495 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.181482 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7fa1a59-2e35-4345-8ab0-e4b18a918a4c" containerName="test-curl-7424-140452070997568" Jul 28 15:23:18.181546 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.181496 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7fa1a59-2e35-4345-8ab0-e4b18a918a4c" containerName="test-curl-7424-140452070997568" Jul 28 15:23:18.181584 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.181568 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7fa1a59-2e35-4345-8ab0-e4b18a918a4c" containerName="test-curl-7424-140452070997568" Jul 28 15:23:18.185982 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.185964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.189261 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.189245 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-aa49c3-openshift-default-dockercfg-dh6kf\"" Jul 28 15:23:18.247577 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.247547 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5"] Jul 28 15:23:18.354737 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.354703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:18.374353 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-data\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374468 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-credential-socket\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374468 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-socket\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374468 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-certs\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374468 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9cp\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-kube-api-access-2k9cp\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374468 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320390ed-6e69-4f99-b0fc-f167b721196b-istiod-ca-cert\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374626 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-istio-token\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374626 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-envoy\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.374626 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.374573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320390ed-6e69-4f99-b0fc-f167b721196b-istio-podinfo\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475141 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-data\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475141 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-credential-socket\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475141 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-socket\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475371 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-certs\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475371 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9cp\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-kube-api-access-2k9cp\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475371 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320390ed-6e69-4f99-b0fc-f167b721196b-istiod-ca-cert\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475371 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-istio-token\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475392 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-envoy\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320390ed-6e69-4f99-b0fc-f167b721196b-istio-podinfo\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475660 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-certs\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475660 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475562 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-socket\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475660 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-credential-socket\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.475660 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.475639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-data\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.476215 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.476196 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320390ed-6e69-4f99-b0fc-f167b721196b-istiod-ca-cert\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.477644 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.477627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-envoy\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.478011 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.477995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320390ed-6e69-4f99-b0fc-f167b721196b-istio-podinfo\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.508719 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.508694 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9cp\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-kube-api-access-2k9cp\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.515296 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.515277 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-istio-token\") pod \"e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.623770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.623631 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:18.798128 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.798099 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-aa49c3-openshift-default-dockercfg-dh6kf\"" Jul 28 15:23:18.805432 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.805404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:18.974514 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:23:18.974489 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320390ed_6e69_4f99_b0fc_f167b721196b.slice/crio-fb4eb081b977625ac83afbae6423e92762eb0452a98bbdace780fe58acc382ca WatchSource:0}: Error finding container fb4eb081b977625ac83afbae6423e92762eb0452a98bbdace780fe58acc382ca: Status 404 returned error can't find the container with id fb4eb081b977625ac83afbae6423e92762eb0452a98bbdace780fe58acc382ca Jul 28 15:23:18.976560 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.976526 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:18.976645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.976595 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:18.976645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.976620 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:18.978721 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:18.978664 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5"] Jul 28 15:23:19.212876 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:19.212804 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:23:19.235376 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:19.235347 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:23:19.935149 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:19.935111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" event={"ID":"320390ed-6e69-4f99-b0fc-f167b721196b","Type":"ContainerStarted","Data":"53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629"} Jul 28 15:23:19.935149 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:19.935158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" event={"ID":"320390ed-6e69-4f99-b0fc-f167b721196b","Type":"ContainerStarted","Data":"fb4eb081b977625ac83afbae6423e92762eb0452a98bbdace780fe58acc382ca"} Jul 28 15:23:20.805957 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:20.805922 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:20.811348 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:20.811324 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:20.882930 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:20.882879 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" podStartSLOduration=2.882866396 podStartE2EDuration="2.882866396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:23:19.975583722 +0000 UTC m=+3901.640825543" watchObservedRunningTime="2026-07-28 15:23:20.882866396 +0000 UTC m=+3902.548108218" Jul 28 15:23:20.940390 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:20.940366 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:20.941082 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:20.941063 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:22.201031 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:22.201003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:22.330116 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:22.330080 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:24.143190 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:24.143160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:24.269863 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:24.269830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:24.710084 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:24.710048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:24.803540 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:24.803507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:31.702211 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:31.702175 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:23:36.878469 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:36.878434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:40.063773 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:40.063741 2562 generic.go:356] "Generic (PLEG): container finished" podID="320390ed-6e69-4f99-b0fc-f167b721196b" containerID="53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629" exitCode=0 Jul 28 15:23:40.064184 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:40.063808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" event={"ID":"320390ed-6e69-4f99-b0fc-f167b721196b","Type":"ContainerDied","Data":"53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629"} Jul 28 15:23:40.064241 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:40.064181 2562 scope.go:117] "RemoveContainer" containerID="53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629" Jul 28 15:23:40.065358 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:40.065324 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:40.065456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:40.065420 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:40.065516 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:40.065456 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:41.071406 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:41.071371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" event={"ID":"320390ed-6e69-4f99-b0fc-f167b721196b","Type":"ContainerStarted","Data":"08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd"} Jul 28 15:23:41.806387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:41.806353 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:41.806791 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:41.806703 2562 patch_prober.go:28] interesting pod/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 28 15:23:41.806989 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:41.806963 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 28 15:23:42.674513 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.674474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:42.806072 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.806032 2562 patch_prober.go:28] interesting pod/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 28 15:23:42.806282 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.806101 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 28 15:23:42.965559 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.965482 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q"] Jul 28 15:23:42.969118 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.969100 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:42.972248 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.972229 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-034645-openshift-default-dockercfg-57zmg\"" Jul 28 15:23:42.990192 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:42.990164 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q"] Jul 28 15:23:43.023699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.023259 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:43.064725 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-socket\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.064725 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-envoy\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.064941 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjj5\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-kube-api-access-crjj5\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.064941 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-credential-socket\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.064941 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-data\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.064941 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ee0591-8b0e-47d8-9695-d78654f79ecc-istiod-ca-cert\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.065084 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.064967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-token\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.065084 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.065050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-certs\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.065084 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.065075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-podinfo\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165613 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-credential-socket\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165613 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-data\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165839 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ee0591-8b0e-47d8-9695-d78654f79ecc-istiod-ca-cert\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165839 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-token\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165913 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-certs\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165947 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-podinfo\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.165995 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.165981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-socket\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166032 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166005 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-data\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166069 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-envoy\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166069 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjj5\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-kube-api-access-crjj5\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166133 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166062 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-credential-socket\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166370 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-socket\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166370 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-certs\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.166544 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.166424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ee0591-8b0e-47d8-9695-d78654f79ecc-istiod-ca-cert\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.168388 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.168358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-envoy\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.168709 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.168686 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-podinfo\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.178981 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.178957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-token\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.179300 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.179280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjj5\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-kube-api-access-crjj5\") pod \"e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.247203 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.247113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:43.281299 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.281271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:43.474385 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.474351 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q"] Jul 28 15:23:43.477826 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:23:43.477795 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ee0591_8b0e_47d8_9695_d78654f79ecc.slice/crio-e65afaf553cab3535df922aade238bb8d32c90eb2fee9b15ef54ec56bf8cd102 WatchSource:0}: Error finding container e65afaf553cab3535df922aade238bb8d32c90eb2fee9b15ef54ec56bf8cd102: Status 404 returned error can't find the container with id e65afaf553cab3535df922aade238bb8d32c90eb2fee9b15ef54ec56bf8cd102 Jul 28 15:23:43.480379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.480002 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:43.480379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.480071 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:43.480379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.480097 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:23:43.805937 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.805892 2562 patch_prober.go:28] interesting pod/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 28 15:23:43.806427 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:43.805964 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 28 15:23:44.089996 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.089900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" event={"ID":"36ee0591-8b0e-47d8-9695-d78654f79ecc","Type":"ContainerStarted","Data":"b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8"} Jul 28 15:23:44.089996 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.089951 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" event={"ID":"36ee0591-8b0e-47d8-9695-d78654f79ecc","Type":"ContainerStarted","Data":"e65afaf553cab3535df922aade238bb8d32c90eb2fee9b15ef54ec56bf8cd102"} Jul 28 15:23:44.116998 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.116935 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podStartSLOduration=2.116915288 podStartE2EDuration="2.116915288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:23:44.115039417 +0000 UTC m=+3925.780281251" watchObservedRunningTime="2026-07-28 15:23:44.116915288 +0000 UTC m=+3925.782157106" Jul 28 15:23:44.282407 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.282371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:44.287196 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.287174 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:44.806651 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.806607 2562 patch_prober.go:28] interesting pod/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 28 15:23:44.807058 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:44.806705 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 28 15:23:45.099107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:45.099029 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:45.100029 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:45.100010 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:23:45.810936 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:45.810905 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:45.811360 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:45.811180 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:45.811770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:45.811750 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:23:46.199039 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:46.198952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:46.300012 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:46.299983 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:49.055538 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:49.055497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:49.332603 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:49.332515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:49.865750 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:49.865720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:49.985829 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:49.985799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:23:53.989756 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:23:53.989717 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:24:01.140809 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:01.140776 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 15:24:01.141186 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:01.140834 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 15:24:02.197267 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:02.197233 2562 generic.go:356] "Generic (PLEG): container finished" podID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerID="b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8" exitCode=0 Jul 28 15:24:02.197902 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:02.197308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" event={"ID":"36ee0591-8b0e-47d8-9695-d78654f79ecc","Type":"ContainerDied","Data":"b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8"} Jul 28 15:24:02.197902 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:02.197806 2562 scope.go:117] "RemoveContainer" containerID="b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8" Jul 28 15:24:02.198864 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:02.198832 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:24:02.198936 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:02.198902 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:24:02.198976 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:02.198934 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:24:03.209011 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:03.208965 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" event={"ID":"36ee0591-8b0e-47d8-9695-d78654f79ecc","Type":"ContainerStarted","Data":"143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef"} Jul 28 15:24:03.282315 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:03.282285 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:24:03.282579 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:03.282551 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 28 15:24:03.282745 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:03.282597 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 28 15:24:04.281872 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:04.281833 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 28 15:24:04.282288 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:04.281886 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 28 15:24:05.282126 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:05.282096 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 28 15:24:05.282511 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:05.282147 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 28 15:24:06.281885 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:06.281849 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 28 15:24:06.282083 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:06.281903 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 28 15:24:06.323977 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:06.323940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:24:07.285901 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:07.285872 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:24:07.286361 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:07.286340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:24:07.287077 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:07.287055 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:24:17.477669 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:17.477581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:24:21.408186 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:21.408149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:24:48.798435 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:48.798404 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:24:58.976269 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:58.976228 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:24:59.291269 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.291163 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl"] Jul 28 15:24:59.294881 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.294864 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.298177 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.298152 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-9498a6-openshift-default-dockercfg-gvng7\"" Jul 28 15:24:59.311156 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.311123 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl"] Jul 28 15:24:59.366123 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.366078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:24:59.447699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-socket\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.447879 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-certs\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.447879 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67fml\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-kube-api-access-67fml\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.447879 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-envoy\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.447982 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447895 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-credential-socket\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.447982 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-data\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.447982 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.447932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0ae42-aadc-4058-acf7-b961e9f49bef-istiod-ca-cert\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.448077 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.448016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-token\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.448077 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.448048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-podinfo\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548475 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548438 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-envoy\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548475 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-credential-socket\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-data\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0ae42-aadc-4058-acf7-b961e9f49bef-istiod-ca-cert\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-token\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-podinfo\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-socket\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548661 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-certs\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.548784 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67fml\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-kube-api-access-67fml\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.549235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.548961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-data\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.549235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.549021 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-credential-socket\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.549235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.549057 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-socket\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.549235 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.549121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-certs\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.549496 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.549476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0ae42-aadc-4058-acf7-b961e9f49bef-istiod-ca-cert\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.550907 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.550885 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-envoy\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.551277 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.551256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-podinfo\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.557004 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.556982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-token\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.557199 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.557176 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67fml\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-kube-api-access-67fml\") pod \"e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.566535 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.566506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:24:59.605563 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.605492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:24:59.751913 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.751886 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl"] Jul 28 15:24:59.754044 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:24:59.754016 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f0ae42_aadc_4058_acf7_b961e9f49bef.slice/crio-9300b6931f42ef0025681b0b7a651947b57787291fbc153c8b15d8f3e866fab5 WatchSource:0}: Error finding container 9300b6931f42ef0025681b0b7a651947b57787291fbc153c8b15d8f3e866fab5: Status 404 returned error can't find the container with id 9300b6931f42ef0025681b0b7a651947b57787291fbc153c8b15d8f3e866fab5 Jul 28 15:24:59.756434 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.756396 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:24:59.756517 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.756484 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:24:59.756558 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:24:59.756526 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:25:00.538841 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:00.538807 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" event={"ID":"33f0ae42-aadc-4058-acf7-b961e9f49bef","Type":"ContainerStarted","Data":"37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4"} Jul 28 15:25:00.539293 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:00.538867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" event={"ID":"33f0ae42-aadc-4058-acf7-b961e9f49bef","Type":"ContainerStarted","Data":"9300b6931f42ef0025681b0b7a651947b57787291fbc153c8b15d8f3e866fab5"} Jul 28 15:25:00.567931 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:00.567868 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" podStartSLOduration=1.5678519 podStartE2EDuration="1.5678519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:25:00.566254452 +0000 UTC m=+4002.231496285" watchObservedRunningTime="2026-07-28 15:25:00.5678519 +0000 UTC m=+4002.233093721" Jul 28 15:25:00.605858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:00.605819 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:00.611215 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:00.611188 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:01.545396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:01.545362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:01.546490 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:01.546472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:02.392238 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:02.392204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:02.626971 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:02.626935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:02.757018 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:02.756922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:06.606809 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:06.606776 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:06.786978 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:06.786945 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:07.225927 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:07.225898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:07.323808 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:07.323773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:14.225230 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:14.225192 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:25:15.057982 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:15.057945 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:15.164135 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:15.164091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:22.490636 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.490540 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:22.670827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.670798 2562 generic.go:356] "Generic (PLEG): container finished" podID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerID="37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4" exitCode=0 Jul 28 15:25:22.670969 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.670856 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" event={"ID":"33f0ae42-aadc-4058-acf7-b961e9f49bef","Type":"ContainerDied","Data":"37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4"} Jul 28 15:25:22.671198 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.671185 2562 scope.go:117] "RemoveContainer" containerID="37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4" Jul 28 15:25:22.678941 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.678905 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:25:22.679057 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.678973 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:25:22.679057 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:22.679004 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 15:25:23.678236 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:23.678204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" event={"ID":"33f0ae42-aadc-4058-acf7-b961e9f49bef","Type":"ContainerStarted","Data":"90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be"} Jul 28 15:25:24.606449 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:24.606412 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:24.606689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:24.606529 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 28 15:25:24.606689 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:24.606569 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 28 15:25:25.607088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:25.606578 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 28 15:25:25.607088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:25.606636 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 28 15:25:26.606849 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:26.606810 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 28 15:25:26.607170 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:26.607133 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 28 15:25:27.610305 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:27.610274 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:27.610806 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:27.610783 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:27.611379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:27.611358 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:35.754443 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:35.754398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:45.061251 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:45.061170 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:45.150256 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:45.150223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:45.250171 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:45.250137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:45.324532 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:45.324448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:45.690898 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:45.690817 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:49.305256 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:49.305222 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:25:52.904123 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:52.904079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:53.015061 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:53.015036 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:53.189058 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:53.188976 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl"] Jul 28 15:25:53.189210 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:53.189174 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" containerID="cri-o://90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be" gracePeriod=30 Jul 28 15:25:53.206065 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:53.206035 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:53.435645 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:53.435617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:56.196952 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:56.196913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:25:57.613025 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:57.612984 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 15:25:57.613425 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:57.613059 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:25:58.440526 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.440500 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:58.558908 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.558883 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-token\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559030 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.558924 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-data\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559030 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.558947 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67fml\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-kube-api-access-67fml\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559030 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559002 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-certs\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559030 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559028 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-podinfo\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559244 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559061 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-socket\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559244 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559098 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-credential-socket\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559244 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559158 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0ae42-aadc-4058-acf7-b961e9f49bef-istiod-ca-cert\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559244 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559181 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-envoy\") pod \"33f0ae42-aadc-4058-acf7-b961e9f49bef\" (UID: \"33f0ae42-aadc-4058-acf7-b961e9f49bef\") " Jul 28 15:25:58.559445 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559287 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:25:58.559606 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559562 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.559606 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559561 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:25:58.559606 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559569 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:25:58.559901 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559825 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-data" (OuterVolumeSpecName: "istio-data") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:25:58.559901 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.559869 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f0ae42-aadc-4058-acf7-b961e9f49bef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:25:58.561144 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.561122 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 15:25:58.561247 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.561192 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:25:58.561397 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.561380 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-token" (OuterVolumeSpecName: "istio-token") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:25:58.561488 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.561391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-kube-api-access-67fml" (OuterVolumeSpecName: "kube-api-access-67fml") pod "33f0ae42-aadc-4058-acf7-b961e9f49bef" (UID: "33f0ae42-aadc-4058-acf7-b961e9f49bef"). InnerVolumeSpecName "kube-api-access-67fml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:25:58.660389 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660356 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0ae42-aadc-4058-acf7-b961e9f49bef-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660389 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660388 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660389 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660396 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660755 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660404 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660755 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660412 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67fml\" (UniqueName: \"kubernetes.io/projected/33f0ae42-aadc-4058-acf7-b961e9f49bef-kube-api-access-67fml\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660755 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660420 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0ae42-aadc-4058-acf7-b961e9f49bef-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660755 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660428 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.660755 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.660436 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0ae42-aadc-4058-acf7-b961e9f49bef-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:25:58.890393 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.890312 2562 generic.go:356] "Generic (PLEG): container finished" podID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerID="90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be" exitCode=0 Jul 28 15:25:58.890393 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.890361 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" Jul 28 15:25:58.890563 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.890389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" event={"ID":"33f0ae42-aadc-4058-acf7-b961e9f49bef","Type":"ContainerDied","Data":"90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be"} Jul 28 15:25:58.890563 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.890443 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl" event={"ID":"33f0ae42-aadc-4058-acf7-b961e9f49bef","Type":"ContainerDied","Data":"9300b6931f42ef0025681b0b7a651947b57787291fbc153c8b15d8f3e866fab5"} Jul 28 15:25:58.890563 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.890467 2562 scope.go:117] "RemoveContainer" containerID="90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be" Jul 28 15:25:58.900237 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.900220 2562 scope.go:117] "RemoveContainer" containerID="37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4" Jul 28 15:25:58.908293 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.908277 2562 scope.go:117] "RemoveContainer" containerID="90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be" Jul 28 15:25:58.908528 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:25:58.908511 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be\": container with ID starting with 90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be not found: ID does not exist" containerID="90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be" Jul 28 15:25:58.908575 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.908535 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be"} err="failed to get container status \"90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be\": rpc error: code = NotFound desc = could not find container \"90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be\": container with ID starting with 90fef3f217f1950d8b831eaf0a8d5ceb81b0ce1a65b4691ec9a95344313ff8be not found: ID does not exist" Jul 28 15:25:58.908575 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.908551 2562 scope.go:117] "RemoveContainer" containerID="37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4" Jul 28 15:25:58.908782 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:25:58.908763 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4\": container with ID starting with 37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4 not found: ID does not exist" containerID="37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4" Jul 28 15:25:58.908853 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.908788 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4"} err="failed to get container status \"37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4\": rpc error: code = NotFound desc = could not find container \"37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4\": container with ID starting with 37792f231ea93894fe20e10cbb3dd8653b96e4405e3dc2de1a50ece311418ac4 not found: ID does not exist" Jul 28 15:25:58.916603 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.916583 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl"] Jul 28 15:25:58.921605 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:25:58.921586 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-9498a6-openshift-default-59cc8c4c66-2ztsl"] Jul 28 15:26:00.888555 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:00.888521 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" path="/var/lib/kubelet/pods/33f0ae42-aadc-4058-acf7-b961e9f49bef/volumes" Jul 28 15:26:13.296807 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:13.296777 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:26:13.542993 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:13.542942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:13.714104 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:13.714022 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:13.809575 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:13.809542 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:13.887529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:13.887495 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:14.271558 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:14.271528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:19.341271 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:19.341230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:23.187398 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:23.187363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:26:55.874814 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:55.874775 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:57.202037 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:57.201993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:58.866601 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:58.866563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:26:58.965654 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:26:58.965624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:03.346917 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:03.346872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:03.424115 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:03.424078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:03.620256 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:03.620177 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5"] Jul 28 15:27:03.620417 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:03.620394 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" containerID="cri-o://08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd" gracePeriod=30 Jul 28 15:27:03.630954 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:03.630928 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:03.900335 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:03.900250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:08.874203 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.874179 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:27:08.986320 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986264 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-credential-socket\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986320 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986312 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320390ed-6e69-4f99-b0fc-f167b721196b-istiod-ca-cert\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986480 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986341 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-envoy\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986480 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986355 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320390ed-6e69-4f99-b0fc-f167b721196b-istio-podinfo\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986480 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986399 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-socket\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986480 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986472 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-data\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986505 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-certs\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986534 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-istio-token\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986561 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k9cp\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-kube-api-access-2k9cp\") pod \"320390ed-6e69-4f99-b0fc-f167b721196b\" (UID: \"320390ed-6e69-4f99-b0fc-f167b721196b\") " Jul 28 15:27:08.986699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986666 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:08.986916 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:08.986916 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986868 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320390ed-6e69-4f99-b0fc-f167b721196b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:27:08.987020 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986932 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:08.987020 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986952 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:08.987020 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.986973 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:08.987386 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.987363 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-data" (OuterVolumeSpecName: "istio-data") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:08.988457 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.988438 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:08.988711 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.988694 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-istio-token" (OuterVolumeSpecName: "istio-token") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:27:08.988835 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.988818 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320390ed-6e69-4f99-b0fc-f167b721196b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 15:27:08.988935 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:08.988891 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-kube-api-access-2k9cp" (OuterVolumeSpecName: "kube-api-access-2k9cp") pod "320390ed-6e69-4f99-b0fc-f167b721196b" (UID: "320390ed-6e69-4f99-b0fc-f167b721196b"). InnerVolumeSpecName "kube-api-access-2k9cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:27:09.088283 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088259 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.088283 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088280 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.088417 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088292 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9cp\" (UniqueName: \"kubernetes.io/projected/320390ed-6e69-4f99-b0fc-f167b721196b-kube-api-access-2k9cp\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.088417 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088304 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.088417 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088316 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320390ed-6e69-4f99-b0fc-f167b721196b-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.088417 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088327 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320390ed-6e69-4f99-b0fc-f167b721196b-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.088417 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.088338 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320390ed-6e69-4f99-b0fc-f167b721196b-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:09.288983 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.288922 2562 generic.go:356] "Generic (PLEG): container finished" podID="320390ed-6e69-4f99-b0fc-f167b721196b" containerID="08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd" exitCode=0 Jul 28 15:27:09.288983 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.288965 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" Jul 28 15:27:09.289114 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.289003 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" event={"ID":"320390ed-6e69-4f99-b0fc-f167b721196b","Type":"ContainerDied","Data":"08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd"} Jul 28 15:27:09.289114 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.289057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5" event={"ID":"320390ed-6e69-4f99-b0fc-f167b721196b","Type":"ContainerDied","Data":"fb4eb081b977625ac83afbae6423e92762eb0452a98bbdace780fe58acc382ca"} Jul 28 15:27:09.289114 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.289085 2562 scope.go:117] "RemoveContainer" containerID="08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd" Jul 28 15:27:09.298848 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.298834 2562 scope.go:117] "RemoveContainer" containerID="53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629" Jul 28 15:27:09.307079 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.307064 2562 scope.go:117] "RemoveContainer" containerID="08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd" Jul 28 15:27:09.307364 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:27:09.307338 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd\": container with ID starting with 08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd not found: ID does not exist" containerID="08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd" Jul 28 15:27:09.307459 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.307370 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd"} err="failed to get container status \"08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd\": rpc error: code = NotFound desc = could not find container \"08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd\": container with ID starting with 08332a1c962a20e8a5d74c75f80ce7dc42021eb1c22f1882daba2736f9ec2ebd not found: ID does not exist" Jul 28 15:27:09.307459 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.307393 2562 scope.go:117] "RemoveContainer" containerID="53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629" Jul 28 15:27:09.307668 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:27:09.307647 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629\": container with ID starting with 53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629 not found: ID does not exist" containerID="53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629" Jul 28 15:27:09.307770 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.307683 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629"} err="failed to get container status \"53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629\": rpc error: code = NotFound desc = could not find container \"53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629\": container with ID starting with 53636099a4805b5715a344038189e0ceb13395afadd43425370f26e24c12a629 not found: ID does not exist" Jul 28 15:27:09.328962 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.328714 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5"] Jul 28 15:27:09.333789 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:09.333756 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-aa49c3-openshift-default-5ccd4b89fb-4xpb5"] Jul 28 15:27:10.889121 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:10.889036 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" path="/var/lib/kubelet/pods/320390ed-6e69-4f99-b0fc-f167b721196b/volumes" Jul 28 15:27:12.430426 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:12.430391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:12.532966 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:12.532932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:12.632498 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:12.632468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:12.710718 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:12.710599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:13.283062 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:13.283026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:16.201519 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:16.201487 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:27:20.230978 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:20.230938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:20.333865 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:20.333830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:20.510398 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:20.510320 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q"] Jul 28 15:27:20.510608 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:20.510581 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" containerID="cri-o://143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef" gracePeriod=30 Jul 28 15:27:20.544622 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:20.544595 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:20.793411 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:20.793321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:22.288386 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:22.288339 2562 patch_prober.go:28] interesting pod/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 15:27:22.288808 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:22.288413 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 15:27:25.759654 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.759630 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:27:25.918873 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.918840 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-data\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.918873 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.918877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-credential-socket\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.918899 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-socket\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.918954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crjj5\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-kube-api-access-crjj5\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.918979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ee0591-8b0e-47d8-9695-d78654f79ecc-istiod-ca-cert\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919075 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919020 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-certs\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919286 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919089 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-token\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919286 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919114 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-envoy\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919286 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919141 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-podinfo\") pod \"36ee0591-8b0e-47d8-9695-d78654f79ecc\" (UID: \"36ee0591-8b0e-47d8-9695-d78654f79ecc\") " Jul 28 15:27:25.919286 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919148 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:25.919551 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919283 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:25.919551 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919350 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:25.919551 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919421 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:25.919551 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919442 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:25.919777 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919592 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ee0591-8b0e-47d8-9695-d78654f79ecc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:27:25.919777 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.919760 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-data" (OuterVolumeSpecName: "istio-data") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:25.921287 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.921258 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:25.921401 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.921321 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 15:27:25.921401 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.921337 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-token" (OuterVolumeSpecName: "istio-token") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:27:25.921401 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:25.921353 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-kube-api-access-crjj5" (OuterVolumeSpecName: "kube-api-access-crjj5") pod "36ee0591-8b0e-47d8-9695-d78654f79ecc" (UID: "36ee0591-8b0e-47d8-9695-d78654f79ecc"). InnerVolumeSpecName "kube-api-access-crjj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:27:26.020270 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020248 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.020270 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020269 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjj5\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-kube-api-access-crjj5\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.020396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020278 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36ee0591-8b0e-47d8-9695-d78654f79ecc-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.020396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020304 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.020396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020312 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.020396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020319 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.020396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.020326 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36ee0591-8b0e-47d8-9695-d78654f79ecc-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:26.387224 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.387185 2562 generic.go:356] "Generic (PLEG): container finished" podID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerID="143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef" exitCode=0 Jul 28 15:27:26.387388 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.387246 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" Jul 28 15:27:26.387388 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.387258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" event={"ID":"36ee0591-8b0e-47d8-9695-d78654f79ecc","Type":"ContainerDied","Data":"143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef"} Jul 28 15:27:26.387388 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.387322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q" event={"ID":"36ee0591-8b0e-47d8-9695-d78654f79ecc","Type":"ContainerDied","Data":"e65afaf553cab3535df922aade238bb8d32c90eb2fee9b15ef54ec56bf8cd102"} Jul 28 15:27:26.387388 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.387360 2562 scope.go:117] "RemoveContainer" containerID="143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef" Jul 28 15:27:26.398215 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.398193 2562 scope.go:117] "RemoveContainer" containerID="b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8" Jul 28 15:27:26.409876 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.409852 2562 scope.go:117] "RemoveContainer" containerID="143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef" Jul 28 15:27:26.410378 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:27:26.410346 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef\": container with ID starting with 143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef not found: ID does not exist" containerID="143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef" Jul 28 15:27:26.410481 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.410384 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef"} err="failed to get container status \"143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef\": rpc error: code = NotFound desc = could not find container \"143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef\": container with ID starting with 143b9eac40a425f562d7b5fc24e8680131e7492e25d0e6ddd961513033c2a7ef not found: ID does not exist" Jul 28 15:27:26.410481 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.410404 2562 scope.go:117] "RemoveContainer" containerID="b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8" Jul 28 15:27:26.410838 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:27:26.410817 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8\": container with ID starting with b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8 not found: ID does not exist" containerID="b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8" Jul 28 15:27:26.410907 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.410844 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8"} err="failed to get container status \"b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8\": rpc error: code = NotFound desc = could not find container \"b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8\": container with ID starting with b764908c56167f20cc7640a4f77d8bb818cec361012943fcef27bee8b94e7fc8 not found: ID does not exist" Jul 28 15:27:26.419296 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.419265 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q"] Jul 28 15:27:26.422856 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.422835 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-034645-openshift-default-7cd7d4f4b4-ntq5q"] Jul 28 15:27:26.888915 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:26.888878 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" path="/var/lib/kubelet/pods/36ee0591-8b0e-47d8-9695-d78654f79ecc/volumes" Jul 28 15:27:28.710953 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:28.710920 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7"] Jul 28 15:27:28.718231 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:28.718206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:28.718570 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:28.718550 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d96a19-29754195-t8ww7"] Jul 28 15:27:28.804863 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:28.804825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:28.889214 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:28.889185 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6cdb21-4194-42f8-a843-419e96095274" path="/var/lib/kubelet/pods/7c6cdb21-4194-42f8-a843-419e96095274/volumes" Jul 28 15:27:28.906162 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:28.906135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:29.006948 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:29.006871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:29.478322 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:29.478286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:31.703525 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:31.703490 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:27:37.470426 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:37.470376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:37.580516 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:37.580485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:37.777573 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:37.777486 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6"] Jul 28 15:27:37.777763 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:37.777737 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" containerID="cri-o://5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5" gracePeriod=30 Jul 28 15:27:37.787699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:37.787660 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:38.047827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.047781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:38.316481 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316416 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl"] Jul 28 15:27:38.316801 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316786 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316802 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316812 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316819 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316832 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316837 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316845 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" Jul 28 15:27:38.316858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316850 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316864 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316869 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316877 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316883 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316935 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316942 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316949 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316957 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ee0591-8b0e-47d8-9695-d78654f79ecc" containerName="istio-proxy" Jul 28 15:27:38.317102 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.316963 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f0ae42-aadc-4058-acf7-b961e9f49bef" containerName="istio-proxy" Jul 28 15:27:38.320072 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.320054 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:38.325574 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.325545 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 15:27:38.325736 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.325581 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cbpmp\"" Jul 28 15:27:38.332381 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.332361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl"] Jul 28 15:27:38.419925 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.419897 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b44b996-1d11-498e-b0a4-5e463ba1775c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl\" (UID: \"3b44b996-1d11-498e-b0a4-5e463ba1775c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:38.520765 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.520741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b44b996-1d11-498e-b0a4-5e463ba1775c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl\" (UID: \"3b44b996-1d11-498e-b0a4-5e463ba1775c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:38.521315 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.521295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b44b996-1d11-498e-b0a4-5e463ba1775c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl\" (UID: \"3b44b996-1d11-498e-b0a4-5e463ba1775c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:38.631140 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.631086 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:38.775316 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:27:38.775287 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b44b996_1d11_498e_b0a4_5e463ba1775c.slice/crio-ca48c8f1d785111e1b1a4d65eb4ef689299d37eb14662b3b109016a6bdcaa8b8 WatchSource:0}: Error finding container ca48c8f1d785111e1b1a4d65eb4ef689299d37eb14662b3b109016a6bdcaa8b8: Status 404 returned error can't find the container with id ca48c8f1d785111e1b1a4d65eb4ef689299d37eb14662b3b109016a6bdcaa8b8 Jul 28 15:27:38.776544 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:38.776517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl"] Jul 28 15:27:39.466242 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:39.466210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" event={"ID":"3b44b996-1d11-498e-b0a4-5e463ba1775c","Type":"ContainerStarted","Data":"f95fa8deca52cd4b29efd795f7069eeab66b8bc4dc09bece902bec54449418a9"} Jul 28 15:27:39.466379 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:39.466259 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" event={"ID":"3b44b996-1d11-498e-b0a4-5e463ba1775c","Type":"ContainerStarted","Data":"ca48c8f1d785111e1b1a4d65eb4ef689299d37eb14662b3b109016a6bdcaa8b8"} Jul 28 15:27:39.496769 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:39.496728 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" podStartSLOduration=1.496712071 podStartE2EDuration="1.496712071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:27:39.495421977 +0000 UTC m=+4161.160663798" watchObservedRunningTime="2026-07-28 15:27:39.496712071 +0000 UTC m=+4161.161953895" Jul 28 15:27:40.473509 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:40.473479 2562 generic.go:356] "Generic (PLEG): container finished" podID="3b44b996-1d11-498e-b0a4-5e463ba1775c" containerID="f95fa8deca52cd4b29efd795f7069eeab66b8bc4dc09bece902bec54449418a9" exitCode=0 Jul 28 15:27:40.473946 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:40.473547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" event={"ID":"3b44b996-1d11-498e-b0a4-5e463ba1775c","Type":"ContainerDied","Data":"f95fa8deca52cd4b29efd795f7069eeab66b8bc4dc09bece902bec54449418a9"} Jul 28 15:27:41.613701 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:41.613660 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:41.745251 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:41.745181 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b44b996-1d11-498e-b0a4-5e463ba1775c-maas-api-service-ca\") pod \"3b44b996-1d11-498e-b0a4-5e463ba1775c\" (UID: \"3b44b996-1d11-498e-b0a4-5e463ba1775c\") " Jul 28 15:27:41.745529 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:41.745505 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b44b996-1d11-498e-b0a4-5e463ba1775c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b44b996-1d11-498e-b0a4-5e463ba1775c" (UID: "3b44b996-1d11-498e-b0a4-5e463ba1775c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:27:41.846547 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:41.846512 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b44b996-1d11-498e-b0a4-5e463ba1775c-maas-api-service-ca\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:42.488785 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:42.488753 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" Jul 28 15:27:42.488969 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:42.488749 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl" event={"ID":"3b44b996-1d11-498e-b0a4-5e463ba1775c","Type":"ContainerDied","Data":"ca48c8f1d785111e1b1a4d65eb4ef689299d37eb14662b3b109016a6bdcaa8b8"} Jul 28 15:27:42.488969 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:42.488887 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca48c8f1d785111e1b1a4d65eb4ef689299d37eb14662b3b109016a6bdcaa8b8" Jul 28 15:27:43.037224 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.037201 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:27:43.055750 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-socket\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.055857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055784 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-certs\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.055857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055806 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-podinfo\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.055857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055831 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaff6d7-097f-442c-ab0b-10aedd31af74-istiod-ca-cert\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.055857 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055846 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-credential-socket\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.056055 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055875 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-token\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.056055 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.055953 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:43.056055 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056003 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-data\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.056055 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056033 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4cgk\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-kube-api-access-q4cgk\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.056212 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056134 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:43.056212 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056147 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:43.056383 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056365 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-certs\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.056456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056389 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-credential-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.056456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056404 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-workload-socket\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.056456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056410 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaff6d7-097f-442c-ab0b-10aedd31af74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 15:27:43.056898 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.056876 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-data" (OuterVolumeSpecName: "istio-data") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:43.057926 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.057900 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 15:27:43.058085 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.058066 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-kube-api-access-q4cgk" (OuterVolumeSpecName: "kube-api-access-q4cgk") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "kube-api-access-q4cgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:27:43.058085 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.058076 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-token" (OuterVolumeSpecName: "istio-token") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 15:27:43.156719 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.156693 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-envoy\") pod \"adaff6d7-097f-442c-ab0b-10aedd31af74\" (UID: \"adaff6d7-097f-442c-ab0b-10aedd31af74\") " Jul 28 15:27:43.156947 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.156931 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-podinfo\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.156947 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.156947 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adaff6d7-097f-442c-ab0b-10aedd31af74-istiod-ca-cert\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.157052 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.156956 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-token\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.157052 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.156963 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-data\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.157052 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.156971 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4cgk\" (UniqueName: \"kubernetes.io/projected/adaff6d7-097f-442c-ab0b-10aedd31af74-kube-api-access-q4cgk\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.158476 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.158455 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "adaff6d7-097f-442c-ab0b-10aedd31af74" (UID: "adaff6d7-097f-442c-ab0b-10aedd31af74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 15:27:43.257978 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.257951 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adaff6d7-097f-442c-ab0b-10aedd31af74-istio-envoy\") on node \"ip-10-0-131-121.ec2.internal\" DevicePath \"\"" Jul 28 15:27:43.496427 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.496361 2562 generic.go:356] "Generic (PLEG): container finished" podID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerID="5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5" exitCode=0 Jul 28 15:27:43.496427 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.496412 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" Jul 28 15:27:43.496554 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.496444 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" event={"ID":"adaff6d7-097f-442c-ab0b-10aedd31af74","Type":"ContainerDied","Data":"5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5"} Jul 28 15:27:43.496554 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.496504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6" event={"ID":"adaff6d7-097f-442c-ab0b-10aedd31af74","Type":"ContainerDied","Data":"b52423822fe65f67a451d9f98c2c87c2f4e520e8efe1bc9fb6c5110b1b7538ff"} Jul 28 15:27:43.496554 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.496538 2562 scope.go:117] "RemoveContainer" containerID="5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5" Jul 28 15:27:43.507103 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.507084 2562 scope.go:117] "RemoveContainer" containerID="c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43" Jul 28 15:27:43.515827 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.515806 2562 scope.go:117] "RemoveContainer" containerID="5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5" Jul 28 15:27:43.516084 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:27:43.516063 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5\": container with ID starting with 5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5 not found: ID does not exist" containerID="5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5" Jul 28 15:27:43.516153 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.516096 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5"} err="failed to get container status \"5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5\": rpc error: code = NotFound desc = could not find container \"5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5\": container with ID starting with 5969b54bb402031c894cc676d68b0bd3f342ee2a46e6fc56242dcbb36d7c17c5 not found: ID does not exist" Jul 28 15:27:43.516153 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.516121 2562 scope.go:117] "RemoveContainer" containerID="c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43" Jul 28 15:27:43.516416 ip-10-0-131-121 kubenswrapper[2562]: E0728 15:27:43.516395 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43\": container with ID starting with c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43 not found: ID does not exist" containerID="c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43" Jul 28 15:27:43.516456 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.516423 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43"} err="failed to get container status \"c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43\": rpc error: code = NotFound desc = could not find container \"c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43\": container with ID starting with c7fedf9b83f295f4ab6b5024f2c8dd62e72d688bfd3f815c034d822a8dc0de43 not found: ID does not exist" Jul 28 15:27:43.524173 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.524145 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6"] Jul 28 15:27:43.526556 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:43.526530 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d96a19-openshift-default-7cf457d644-2xdp6"] Jul 28 15:27:44.888895 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:44.888863 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" path="/var/lib/kubelet/pods/adaff6d7-097f-442c-ab0b-10aedd31af74/volumes" Jul 28 15:27:44.896338 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:44.896312 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z"] Jul 28 15:27:44.898255 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:44.898214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:44.901345 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:44.901323 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3edd55-29754195-kvm2z"] Jul 28 15:27:44.994841 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:44.994810 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:45.079420 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:45.079395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:45.203858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:45.203781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:45.205425 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:45.205396 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl"] Jul 28 15:27:45.207431 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:45.207412 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-3edd55-cgzcl"] Jul 28 15:27:45.576699 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:45.576648 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:46.889527 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:46.889488 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b44b996-1d11-498e-b0a4-5e463ba1775c" path="/var/lib/kubelet/pods/3b44b996-1d11-498e-b0a4-5e463ba1775c/volumes" Jul 28 15:27:46.889967 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:46.889807 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ba50e5-ac74-4c0d-90f3-3f1255456639" path="/var/lib/kubelet/pods/a0ba50e5-ac74-4c0d-90f3-3f1255456639/volumes" Jul 28 15:27:47.899430 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:47.899396 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6h9fn"] Jul 28 15:27:52.792628 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:52.792587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:52.890360 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:52.890312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:53.119383 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:53.119351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:53.359332 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:53.359299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:27:59.521166 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:59.521124 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-br66k_ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1/manager/0.log" Jul 28 15:27:59.646282 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:27:59.646247 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7mgvp_82933eb6-2f55-41b8-9829-099ea0f50185/manager/0.log" Jul 28 15:28:00.427553 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:00.427525 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f7956c87-wdnp5_52fff165-aeb1-4ef7-b579-1016e42c6293/copy-manifests/0.log" Jul 28 15:28:00.438040 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:00.438014 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f7956c87-wdnp5_52fff165-aeb1-4ef7-b579-1016e42c6293/manager/0.log" Jul 28 15:28:00.934500 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:00.934475 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754165-d6b2t_db72740a-174a-426e-9c86-6644b275a578/cleanup/0.log" Jul 28 15:28:01.057629 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:01.057605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754180-5twkk_336aa749-62e1-4351-b83b-c61dc378c8dd/cleanup/0.log" Jul 28 15:28:01.168793 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:01.168770 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754195-vm8vd_608eb65f-2368-47e8-b1b9-69efafdc0ba2/cleanup/0.log" Jul 28 15:28:02.086683 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.086651 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6_db07b740-8d8a-4129-87c2-b65d291b8c2b/util/0.log" Jul 28 15:28:02.094840 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.094815 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6_db07b740-8d8a-4129-87c2-b65d291b8c2b/pull/0.log" Jul 28 15:28:02.101887 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.101860 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6_db07b740-8d8a-4129-87c2-b65d291b8c2b/extract/0.log" Jul 28 15:28:02.217088 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.217067 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt_da195d31-8d76-4240-b6d0-929aeaac932d/util/0.log" Jul 28 15:28:02.226255 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.226230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt_da195d31-8d76-4240-b6d0-929aeaac932d/pull/0.log" Jul 28 15:28:02.239539 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.239518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt_da195d31-8d76-4240-b6d0-929aeaac932d/extract/0.log" Jul 28 15:28:02.361579 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.361512 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q_d4cd38f1-a086-4342-9f30-5115bc0b4f5e/util/0.log" Jul 28 15:28:02.370161 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.370139 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q_d4cd38f1-a086-4342-9f30-5115bc0b4f5e/pull/0.log" Jul 28 15:28:02.380618 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.380598 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q_d4cd38f1-a086-4342-9f30-5115bc0b4f5e/extract/0.log" Jul 28 15:28:02.498825 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.498805 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8_f8259e71-b3f1-4ce3-bc6a-1708759f8df1/util/0.log" Jul 28 15:28:02.506958 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.506933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8_f8259e71-b3f1-4ce3-bc6a-1708759f8df1/pull/0.log" Jul 28 15:28:02.516927 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.516878 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8_f8259e71-b3f1-4ce3-bc6a-1708759f8df1/extract/0.log" Jul 28 15:28:02.777525 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.777432 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tztkf_5e0522f1-d306-4ba2-82b9-386ff956491e/manager/0.log" Jul 28 15:28:02.902876 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:02.902853 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-54dfm_8798c72d-80b7-4216-ac70-3c02639da1e9/manager/0.log" Jul 28 15:28:03.015961 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:03.015938 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-df78m_3fa1a911-90cb-491c-979c-140e3b2873d3/kuadrant-console-plugin/0.log" Jul 28 15:28:03.397934 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:03.397907 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6h9fn_fa9894ff-4367-4136-b8c7-083562331b78/limitador/0.log" Jul 28 15:28:04.774903 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:04.774870 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77859b6fc7-vhkkx_648fe082-e7c4-40bf-a820-b594afd97f9c/router/0.log" Jul 28 15:28:05.498400 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:05.498369 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fw69p9_167625da-ba48-49ce-9a4d-647f73df47bb/main/0.log" Jul 28 15:28:11.597007 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.596972 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hzqjc/must-gather-pbnfq"] Jul 28 15:28:11.597387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597286 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" Jul 28 15:28:11.597387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597297 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" Jul 28 15:28:11.597387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597325 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b44b996-1d11-498e-b0a4-5e463ba1775c" containerName="revoke-keys" Jul 28 15:28:11.597387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597330 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b44b996-1d11-498e-b0a4-5e463ba1775c" containerName="revoke-keys" Jul 28 15:28:11.597387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597382 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="320390ed-6e69-4f99-b0fc-f167b721196b" containerName="istio-proxy" Jul 28 15:28:11.597387 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597391 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" Jul 28 15:28:11.597617 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597398 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b44b996-1d11-498e-b0a4-5e463ba1775c" containerName="revoke-keys" Jul 28 15:28:11.597617 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597485 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" Jul 28 15:28:11.597617 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597491 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" Jul 28 15:28:11.597617 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.597547 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="adaff6d7-097f-442c-ab0b-10aedd31af74" containerName="istio-proxy" Jul 28 15:28:11.600464 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.600446 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.603551 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.603524 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hzqjc\"/\"default-dockercfg-vtfl4\"" Jul 28 15:28:11.604428 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.604409 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzqjc\"/\"kube-root-ca.crt\"" Jul 28 15:28:11.604911 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.604893 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hzqjc\"/\"openshift-service-ca.crt\"" Jul 28 15:28:11.612726 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.612700 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzqjc/must-gather-pbnfq"] Jul 28 15:28:11.617190 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.617165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:28:11.672182 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.672161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d846f77-d5ec-4a86-bbdf-864746d8c7bd-must-gather-output\") pod \"must-gather-pbnfq\" (UID: \"3d846f77-d5ec-4a86-bbdf-864746d8c7bd\") " pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.672270 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.672209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2f8\" (UniqueName: \"kubernetes.io/projected/3d846f77-d5ec-4a86-bbdf-864746d8c7bd-kube-api-access-zm2f8\") pod \"must-gather-pbnfq\" (UID: \"3d846f77-d5ec-4a86-bbdf-864746d8c7bd\") " pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.773095 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.773067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d846f77-d5ec-4a86-bbdf-864746d8c7bd-must-gather-output\") pod \"must-gather-pbnfq\" (UID: \"3d846f77-d5ec-4a86-bbdf-864746d8c7bd\") " pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.773209 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.773153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2f8\" (UniqueName: \"kubernetes.io/projected/3d846f77-d5ec-4a86-bbdf-864746d8c7bd-kube-api-access-zm2f8\") pod \"must-gather-pbnfq\" (UID: \"3d846f77-d5ec-4a86-bbdf-864746d8c7bd\") " pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.773396 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.773374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d846f77-d5ec-4a86-bbdf-864746d8c7bd-must-gather-output\") pod \"must-gather-pbnfq\" (UID: \"3d846f77-d5ec-4a86-bbdf-864746d8c7bd\") " pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.786174 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.786150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2f8\" (UniqueName: \"kubernetes.io/projected/3d846f77-d5ec-4a86-bbdf-864746d8c7bd-kube-api-access-zm2f8\") pod \"must-gather-pbnfq\" (UID: \"3d846f77-d5ec-4a86-bbdf-864746d8c7bd\") " pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:11.911428 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:11.911360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzqjc/must-gather-pbnfq" Jul 28 15:28:12.054751 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:12.054730 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzqjc/must-gather-pbnfq"] Jul 28 15:28:12.055781 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:28:12.055755 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d846f77_d5ec_4a86_bbdf_864746d8c7bd.slice/crio-bcc36592337c7dcee9849b4ba2a33e08ad0575fb78122ba1796e87171661f9b3 WatchSource:0}: Error finding container bcc36592337c7dcee9849b4ba2a33e08ad0575fb78122ba1796e87171661f9b3: Status 404 returned error can't find the container with id bcc36592337c7dcee9849b4ba2a33e08ad0575fb78122ba1796e87171661f9b3 Jul 28 15:28:12.058069 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:12.058052 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 15:28:12.668316 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:12.668281 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzqjc/must-gather-pbnfq" event={"ID":"3d846f77-d5ec-4a86-bbdf-864746d8c7bd","Type":"ContainerStarted","Data":"bcc36592337c7dcee9849b4ba2a33e08ad0575fb78122ba1796e87171661f9b3"} Jul 28 15:28:13.280886 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:13.280785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" limit=1024 Jul 28 15:28:13.679375 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:13.679331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzqjc/must-gather-pbnfq" event={"ID":"3d846f77-d5ec-4a86-bbdf-864746d8c7bd","Type":"ContainerStarted","Data":"c2d404ed47800226f21cc51ddfa869f198fb06d5dd481a3d8393b9e19e0c715f"} Jul 28 15:28:13.679815 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:13.679389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzqjc/must-gather-pbnfq" event={"ID":"3d846f77-d5ec-4a86-bbdf-864746d8c7bd","Type":"ContainerStarted","Data":"cf369aa819cb0f3f7f6a58aea73178d27c0f1d63d9d7bbecacc7b7d9857a7cb4"} Jul 28 15:28:13.705298 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:13.705251 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hzqjc/must-gather-pbnfq" podStartSLOduration=1.921537974 podStartE2EDuration="2.705236167s" totalImagesPullingTime="783.698193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 15:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:28:13.703993387 +0000 UTC m=+4195.369235208" watchObservedRunningTime="2026-07-28 15:28:13.705236167 +0000 UTC m=+4195.370477988" Jul 28 15:28:14.827077 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:14.827041 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fgmfx_5a38c15d-bd62-4e10-8c87-120662bd95f7/global-pull-secret-syncer/0.log" Jul 28 15:28:14.913947 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:14.913920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bb4bs_dc3f2bf9-a419-4e40-a236-ca08eabe03bb/konnectivity-agent/0.log" Jul 28 15:28:15.137284 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:15.137212 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-121.ec2.internal_0c5d5894bca76db08e7d990578b15cf0/haproxy/0.log" Jul 28 15:28:19.285311 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:19.285230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:28:19.302639 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:19.302614 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brkdl_17ca5711-b9a9-4b33-b8e8-cecf9dc090f4/ovn-acl-logging/0.log" Jul 28 15:28:19.505582 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:19.505553 2562 scope.go:117] "RemoveContainer" containerID="49bf7869791f04662c4550d74ccfb03974a07d465feb39152e8bed1d91d48fcf" Jul 28 15:28:19.518216 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:19.518129 2562 scope.go:117] "RemoveContainer" containerID="f6161683eda193c92f1f833ebe4114197756ced45ed3831c2f772e551cc6dd01" Jul 28 15:28:19.998103 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:19.998006 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6_db07b740-8d8a-4129-87c2-b65d291b8c2b/extract/0.log" Jul 28 15:28:20.053695 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.053640 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6_db07b740-8d8a-4129-87c2-b65d291b8c2b/util/0.log" Jul 28 15:28:20.112337 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.112305 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759766h6_db07b740-8d8a-4129-87c2-b65d291b8c2b/pull/0.log" Jul 28 15:28:20.181329 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.181284 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt_da195d31-8d76-4240-b6d0-929aeaac932d/extract/0.log" Jul 28 15:28:20.267685 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.267591 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt_da195d31-8d76-4240-b6d0-929aeaac932d/util/0.log" Jul 28 15:28:20.330568 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.330535 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0f2dbt_da195d31-8d76-4240-b6d0-929aeaac932d/pull/0.log" Jul 28 15:28:20.393513 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.393475 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q_d4cd38f1-a086-4342-9f30-5115bc0b4f5e/extract/0.log" Jul 28 15:28:20.447664 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.447629 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q_d4cd38f1-a086-4342-9f30-5115bc0b4f5e/util/0.log" Jul 28 15:28:20.511993 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.511964 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nxv7q_d4cd38f1-a086-4342-9f30-5115bc0b4f5e/pull/0.log" Jul 28 15:28:20.556407 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.556375 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8_f8259e71-b3f1-4ce3-bc6a-1708759f8df1/extract/0.log" Jul 28 15:28:20.600631 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.600593 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8_f8259e71-b3f1-4ce3-bc6a-1708759f8df1/util/0.log" Jul 28 15:28:20.643907 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:20.643875 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1fxgx8_f8259e71-b3f1-4ce3-bc6a-1708759f8df1/pull/0.log" Jul 28 15:28:21.434731 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:21.434619 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tztkf_5e0522f1-d306-4ba2-82b9-386ff956491e/manager/0.log" Jul 28 15:28:21.501225 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:21.501176 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-54dfm_8798c72d-80b7-4216-ac70-3c02639da1e9/manager/0.log" Jul 28 15:28:21.561653 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:21.561611 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-df78m_3fa1a911-90cb-491c-979c-140e3b2873d3/kuadrant-console-plugin/0.log" Jul 28 15:28:22.100044 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:22.100015 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6h9fn_fa9894ff-4367-4136-b8c7-083562331b78/limitador/0.log" Jul 28 15:28:23.569947 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.569908 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr"] Jul 28 15:28:23.575395 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.575367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.605582 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.605545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr"] Jul 28 15:28:23.699173 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.699136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-lib-modules\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.699373 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.699183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphwm\" (UniqueName: \"kubernetes.io/projected/7761aa19-f403-40c3-acc8-f09fba2d40fa-kube-api-access-nphwm\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.699373 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.699287 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-podres\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.699373 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.699359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-proc\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.699552 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.699406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-sys\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800263 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-proc\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800463 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-sys\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800463 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-lib-modules\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800463 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800349 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-proc\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800463 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphwm\" (UniqueName: \"kubernetes.io/projected/7761aa19-f403-40c3-acc8-f09fba2d40fa-kube-api-access-nphwm\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800463 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800412 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-sys\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800463 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-podres\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800778 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800504 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-podres\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.800778 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.800509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7761aa19-f403-40c3-acc8-f09fba2d40fa-lib-modules\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.810573 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.810547 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphwm\" (UniqueName: \"kubernetes.io/projected/7761aa19-f403-40c3-acc8-f09fba2d40fa-kube-api-access-nphwm\") pod \"perf-node-gather-daemonset-q45lr\" (UID: \"7761aa19-f403-40c3-acc8-f09fba2d40fa\") " pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.888910 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.888840 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:23.954152 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:23.952560 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-9x8ng_0dba08a3-6d95-4cf5-aaf6-6f4549eb1531/cluster-monitoring-operator/0.log" Jul 28 15:28:24.024408 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.024382 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-btszk_db4b61fd-73ee-4028-9f67-d291563e3e28/kube-state-metrics/0.log" Jul 28 15:28:24.078782 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.078751 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-btszk_db4b61fd-73ee-4028-9f67-d291563e3e28/kube-rbac-proxy-main/0.log" Jul 28 15:28:24.089958 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.089930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr"] Jul 28 15:28:24.090990 ip-10-0-131-121 kubenswrapper[2562]: W0728 15:28:24.090963 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7761aa19_f403_40c3_acc8_f09fba2d40fa.slice/crio-6d4d5c59d2aced30d80ee7ca3adea97747f50b9b13a620eb82d2e52db5a3d5e9 WatchSource:0}: Error finding container 6d4d5c59d2aced30d80ee7ca3adea97747f50b9b13a620eb82d2e52db5a3d5e9: Status 404 returned error can't find the container with id 6d4d5c59d2aced30d80ee7ca3adea97747f50b9b13a620eb82d2e52db5a3d5e9 Jul 28 15:28:24.133068 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.132962 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-btszk_db4b61fd-73ee-4028-9f67-d291563e3e28/kube-rbac-proxy-self/0.log" Jul 28 15:28:24.225359 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.225329 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-p26sm_555011e8-077d-4f1b-a46f-06212830bef4/monitoring-plugin/0.log" Jul 28 15:28:24.272973 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.272942 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftxr8_2a4434f1-2784-4ba2-9f64-0742cc2429db/node-exporter/0.log" Jul 28 15:28:24.305180 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.305147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftxr8_2a4434f1-2784-4ba2-9f64-0742cc2429db/kube-rbac-proxy/0.log" Jul 28 15:28:24.349591 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.349563 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ftxr8_2a4434f1-2784-4ba2-9f64-0742cc2429db/init-textfile/0.log" Jul 28 15:28:24.772384 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.772347 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" event={"ID":"7761aa19-f403-40c3-acc8-f09fba2d40fa","Type":"ContainerStarted","Data":"ba0c26ca7118865f61f407e5afa05ff63b52e4c535008f63b322d090ce7effa6"} Jul 28 15:28:24.772810 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.772410 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" event={"ID":"7761aa19-f403-40c3-acc8-f09fba2d40fa","Type":"ContainerStarted","Data":"6d4d5c59d2aced30d80ee7ca3adea97747f50b9b13a620eb82d2e52db5a3d5e9"} Jul 28 15:28:24.772810 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.772468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:24.802018 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:24.801955 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" podStartSLOduration=1.801933469 podStartE2EDuration="1.801933469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 15:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 15:28:24.799558195 +0000 UTC m=+4206.464800018" watchObservedRunningTime="2026-07-28 15:28:24.801933469 +0000 UTC m=+4206.467175293" Jul 28 15:28:25.038500 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:25.038427 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-mlmrc_b61d58bd-e22c-4535-b49d-96a879e8b042/prometheus-operator/0.log" Jul 28 15:28:25.066131 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:25.066104 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-mlmrc_b61d58bd-e22c-4535-b49d-96a879e8b042/kube-rbac-proxy/0.log" Jul 28 15:28:26.568027 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:26.567991 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-p9h77_4b586543-0933-4f68-8be0-fce454e73554/networking-console-plugin/0.log" Jul 28 15:28:27.633183 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:27.633155 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bcb87dbb9-9bl5s_c22c4c7e-c5f5-4ab7-a402-c7a16b550525/console/0.log" Jul 28 15:28:29.408651 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:29.408624 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qlhl_6f183322-bfbb-4763-b013-72e012cfa318/dns/0.log" Jul 28 15:28:29.436992 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:29.436968 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7qlhl_6f183322-bfbb-4763-b013-72e012cfa318/kube-rbac-proxy/0.log" Jul 28 15:28:29.581741 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:29.581671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bs7j4_f842320b-70c6-466f-9e1c-fcf38d578666/dns-node-resolver/0.log" Jul 28 15:28:30.351477 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:30.351452 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ttdk2_a4f710d1-1eaa-45e8-8853-4bdc020989a0/node-ca/0.log" Jul 28 15:28:30.791667 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:30.791577 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hzqjc/perf-node-gather-daemonset-q45lr" Jul 28 15:28:31.780901 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:31.780863 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77859b6fc7-vhkkx_648fe082-e7c4-40bf-a820-b594afd97f9c/router/0.log" Jul 28 15:28:32.917783 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:32.917697 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vt9w5_7cc40c6d-de39-4283-bb66-f6aa7bb99ac0/serve-healthcheck-canary/0.log" Jul 28 15:28:33.598846 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:33.598822 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-qml4d_e16a89f6-30f6-4e0d-9823-df204fb02fdd/insights-operator/0.log" Jul 28 15:28:33.602621 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:33.602593 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-qml4d_e16a89f6-30f6-4e0d-9823-df204fb02fdd/insights-operator/1.log" Jul 28 15:28:33.717858 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:33.717782 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bm8mr_6aec9c3a-0460-4712-b808-9665f7eb3d39/kube-rbac-proxy/0.log" Jul 28 15:28:33.742504 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:33.742479 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bm8mr_6aec9c3a-0460-4712-b808-9665f7eb3d39/exporter/0.log" Jul 28 15:28:33.768326 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:33.768301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bm8mr_6aec9c3a-0460-4712-b808-9665f7eb3d39/extractor/0.log" Jul 28 15:28:34.688912 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:34.688887 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77859b6fc7-vhkkx_648fe082-e7c4-40bf-a820-b594afd97f9c/router/0.log" Jul 28 15:28:36.292107 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:36.292080 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-77859b6fc7-vhkkx_648fe082-e7c4-40bf-a820-b594afd97f9c/router/0.log" Jul 28 15:28:36.386281 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:36.386249 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-br66k_ef2e92d3-7ea2-4ab6-a00f-ce103e4b1bf1/manager/0.log" Jul 28 15:28:36.825914 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:36.825880 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7mgvp_82933eb6-2f55-41b8-9829-099ea0f50185/manager/0.log" Jul 28 15:28:37.425872 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:37.425847 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f7956c87-wdnp5_52fff165-aeb1-4ef7-b579-1016e42c6293/manager/0.log" Jul 28 15:28:37.467333 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:37.467308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65f7956c87-wdnp5_52fff165-aeb1-4ef7-b579-1016e42c6293/copy-manifests/0.log" Jul 28 15:28:38.595826 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:38.595793 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d8c94b858-p9br9_773f393a-bdd0-4246-90f8-1610375f05be/manager/0.log" Jul 28 15:28:38.689922 ip-10-0-131-121 kubenswrapper[2562]: I0728 15:28:38.689898 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hx828_580b1b4d-db98-4b1f-9d3b-66f4d485c9c2/openshift-lws-operator/0.log"