Jul 24 15:08:04.658942 ip-10-0-132-115 systemd[1]: Starting Kubernetes Kubelet... Jul 24 15:08:05.076902 ip-10-0-132-115 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:08:05.076902 ip-10-0-132-115 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 15:08:05.076902 ip-10-0-132-115 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:08:05.076902 ip-10-0-132-115 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 15:08:05.076902 ip-10-0-132-115 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:08:05.078458 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.078358 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 15:08:05.084780 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084753 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:05.084780 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084775 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:05.084780 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084779 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:05.084780 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084782 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:05.084780 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084786 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:05.084780 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084789 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084796 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084799 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084802 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084805 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084808 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084812 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084815 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084818 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084820 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084824 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084827 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084829 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084832 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084837 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084840 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084843 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084845 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084848 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084851 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:05.085022 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084853 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084856 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084859 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084861 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084864 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084869 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084876 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084879 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084882 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084885 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084890 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084894 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084897 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084899 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084904 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084906 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084909 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084912 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084917 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:05.085489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084920 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084923 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084925 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084928 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084931 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084933 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084936 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084939 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084942 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084945 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084947 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084952 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084955 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084957 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084960 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084963 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084966 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084969 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084971 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084974 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:05.086031 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084977 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084980 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084982 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084985 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084990 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084993 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.084997 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085001 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085003 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085006 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085009 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085012 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085014 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085017 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085020 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085023 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085028 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085031 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085033 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085038 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:05.086514 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085041 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085044 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085047 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085050 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085053 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085056 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085059 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085062 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085067 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085070 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085072 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085075 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085078 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085080 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.085083 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086611 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086621 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086624 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086628 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086631 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:05.087021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086634 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086636 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086642 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086645 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086648 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086651 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086654 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086657 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086659 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086662 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086665 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086669 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086673 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086676 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086682 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086685 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086688 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086691 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086694 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:05.087489 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086696 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086699 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086702 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086705 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086707 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086710 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086713 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086716 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086721 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086724 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086727 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086730 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086732 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086736 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086739 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086742 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086744 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086747 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086750 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086753 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:05.087990 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086758 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086761 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086764 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086767 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086770 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086773 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086775 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086779 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086782 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086785 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086787 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086790 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086795 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086798 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086801 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086804 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086806 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086809 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086812 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086815 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:05.088494 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086817 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086820 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086823 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086827 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086833 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086836 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086838 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086841 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086844 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086846 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086849 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086852 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086854 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086858 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086860 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086863 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086870 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086874 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086891 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086898 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:05.088991 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086901 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086903 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086906 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086909 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086911 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086914 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086917 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086920 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086922 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086927 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086930 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086932 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086935 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086938 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.086941 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087544 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087549 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087555 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087558 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087561 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:05.089470 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087564 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087566 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087569 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087572 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087575 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087578 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087581 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087584 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087587 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087592 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087595 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087614 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087617 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087620 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087623 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087625 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087628 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087631 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087634 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087636 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:05.089973 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087639 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087644 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087647 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087651 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087653 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087656 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087658 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087661 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087664 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087668 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087671 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087674 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087677 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087683 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087685 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087688 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087690 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087693 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087698 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:05.090442 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087702 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087706 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087709 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087712 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087716 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087719 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087724 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087727 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087729 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087732 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087735 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087737 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087740 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087743 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087745 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087748 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087750 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087754 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087759 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:05.090944 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087762 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087764 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087767 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087770 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087772 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087775 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087778 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087780 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087783 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087786 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087789 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087795 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087800 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087803 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087805 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087808 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087811 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087814 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087817 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087819 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:05.091390 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087822 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087824 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087827 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087830 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087835 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087837 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087840 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087843 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087845 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087848 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087850 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087853 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087855 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087858 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087861 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.087863 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088247 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088263 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088273 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088280 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088286 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 15:08:05.091880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088292 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088299 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088305 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088310 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088314 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088323 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088331 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088336 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088343 2563 flags.go:64] FLAG: --cgroup-root="" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088347 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088354 2563 flags.go:64] FLAG: --client-ca-file="" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088360 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088365 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088376 2563 flags.go:64] FLAG: --cluster-domain="" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088381 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088388 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088393 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088401 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088412 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088417 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088422 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088425 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088429 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088431 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 15:08:05.092380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088434 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088438 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088441 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088446 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088450 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088453 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088456 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088460 2563 flags.go:64] FLAG: --enable-server="true" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088463 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088469 2563 flags.go:64] FLAG: --event-burst="100" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088472 2563 flags.go:64] FLAG: --event-qps="50" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088476 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088479 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088482 2563 flags.go:64] FLAG: --eviction-hard="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088491 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088494 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088497 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088501 2563 flags.go:64] FLAG: --eviction-soft="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088504 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088507 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088511 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088514 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088517 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088520 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088523 2563 flags.go:64] FLAG: --feature-gates="" Jul 24 15:08:05.092988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088527 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088530 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088533 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088536 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088539 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088542 2563 flags.go:64] FLAG: --help="false" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088545 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088549 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088552 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088555 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088562 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088566 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088569 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088572 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088575 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088578 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088581 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088584 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088586 2563 flags.go:64] FLAG: --kube-reserved="" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088589 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088592 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088595 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088598 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088614 2563 flags.go:64] FLAG: --lock-file="" Jul 24 15:08:05.093619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088618 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088622 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088625 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088631 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088635 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088638 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088641 2563 flags.go:64] FLAG: --logging-format="text" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088644 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088647 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088650 2563 flags.go:64] FLAG: --manifest-url="" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088653 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088657 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088661 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088665 2563 flags.go:64] FLAG: --max-pods="110" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088668 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088671 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088674 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088677 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088682 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088685 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088688 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088696 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088699 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088703 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 15:08:05.094233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088706 2563 flags.go:64] FLAG: --pod-cidr="" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088709 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088714 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088739 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088743 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088746 2563 flags.go:64] FLAG: --port="10250" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088749 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088752 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-003c5591f41240239" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088755 2563 flags.go:64] FLAG: --qos-reserved="" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088758 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088761 2563 flags.go:64] FLAG: --register-node="true" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088764 2563 flags.go:64] FLAG: --register-with-taints="" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088768 2563 flags.go:64] FLAG: --registry-burst="10" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088772 2563 flags.go:64] FLAG: --registry-qps="5" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088774 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088777 2563 flags.go:64] FLAG: --reserved-memory="" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088781 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088784 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088787 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088790 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088793 2563 flags.go:64] FLAG: --runonce="false" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088796 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088799 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088802 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088804 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 15:08:05.094813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088807 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088812 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088815 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088818 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088821 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088824 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088827 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088830 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088832 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088836 2563 flags.go:64] FLAG: --system-cgroups="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088838 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088844 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088847 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088849 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088854 2563 flags.go:64] FLAG: --tls-min-version="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088856 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088859 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088862 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088865 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088868 2563 flags.go:64] FLAG: --v="2" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088872 2563 flags.go:64] FLAG: --version="false" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088877 2563 flags.go:64] FLAG: --vmodule="" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088881 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.088885 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089015 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:05.095400 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089019 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089022 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089025 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089028 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089031 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089034 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089036 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089039 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089044 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089047 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089051 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089054 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089056 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089059 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089062 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089065 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089067 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089070 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089072 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089075 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:05.096066 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089077 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089080 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089083 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089086 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089090 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089093 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089096 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089099 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089102 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089105 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089109 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089111 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089114 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089117 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089120 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089122 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089125 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089127 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089130 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:05.096546 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089132 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089135 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089139 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089142 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089144 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089147 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089149 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089152 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089155 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089157 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089160 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089162 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089165 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089168 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089170 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089173 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089176 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089178 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089181 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089183 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:05.097012 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089186 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089188 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089192 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089196 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089198 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089201 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089204 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089206 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089209 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089211 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089213 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089216 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089218 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089221 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089225 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089228 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089230 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089233 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089235 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089238 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:05.097487 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089240 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089243 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089246 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089248 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089251 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089253 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089256 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089258 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089261 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089264 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089266 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089269 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089271 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089274 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089276 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089279 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089282 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089284 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:05.097999 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.089287 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.089851 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.096979 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.096997 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.097756 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.097767 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097831 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097837 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097842 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097847 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097850 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097853 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097856 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097859 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097862 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:05.098446 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097865 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097867 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097870 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097872 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097875 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097878 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097881 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097883 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097886 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097888 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097891 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097893 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097895 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097898 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097900 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097903 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097906 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097909 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097911 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097914 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:05.098845 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097919 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097922 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097936 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097939 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097943 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097947 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097950 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097953 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097955 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097958 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097960 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097964 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097968 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097970 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097973 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097976 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097979 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097982 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097985 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:05.099328 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097987 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097990 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097992 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097995 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.097997 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098000 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098003 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098006 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098008 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098011 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098014 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098017 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098019 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098022 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098026 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098029 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098031 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098034 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098036 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098039 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:05.099824 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098041 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098044 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098046 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098049 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098052 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098054 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098056 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098059 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098061 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098064 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098066 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098069 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098071 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098074 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098077 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098079 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098081 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098084 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098087 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098089 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:05.100300 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098091 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098094 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098096 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098098 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098101 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098104 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098107 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098110 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098113 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098116 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098118 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.098124 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098281 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098288 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:05.100785 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098292 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098296 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098299 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098302 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098305 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098308 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098310 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098313 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098315 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098318 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098321 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098323 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098325 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098328 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098331 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098333 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098335 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098338 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098340 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098343 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:05.101130 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098346 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098348 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098351 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098353 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098356 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098359 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098362 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098365 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098367 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098370 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098372 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098375 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098377 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098380 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098382 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098385 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098387 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098390 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098393 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098395 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:05.101595 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098398 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098400 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098403 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098405 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098408 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098411 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098415 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098418 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098420 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098423 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098425 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098428 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098431 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098434 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098436 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098439 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098442 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098445 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098447 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098451 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:05.102129 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098454 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098457 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098460 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098463 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098466 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098468 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098471 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098473 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098476 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098479 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098481 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098484 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098486 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098489 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098491 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098494 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098496 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098499 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098501 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:05.102619 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098504 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098506 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098509 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098511 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098513 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098518 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098521 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098523 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098526 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098529 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098532 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098534 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098537 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098540 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098543 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098547 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098550 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:05.103177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:05.098553 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:05.103738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.098558 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:08:05.103738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.098704 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 15:08:05.103738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.101535 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 15:08:05.103738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.102915 2563 server.go:1019] "Starting client certificate rotation" Jul 24 15:08:05.103738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.103034 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:08:05.103738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.103088 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:08:05.105544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.105522 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:08:05.107957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.107920 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:08:05.121341 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.121321 2563 log.go:25] "Validated CRI v1 runtime API" Jul 24 15:08:05.124531 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.124505 2563 log.go:25] "Validated CRI v1 image API" Jul 24 15:08:05.125834 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.125809 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 15:08:05.133462 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.133435 2563 fs.go:135] Filesystem UUIDs: map[23610412-d094-4511-9f9d-795eb3a49205:/dev/nvme0n1p3 3a3c0487-bd90-48ac-a658-38aa4e12f091:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 24 15:08:05.133569 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.133458 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 15:08:05.133993 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.133974 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:08:05.138490 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.138375 2563 manager.go:216] Machine: {Timestamp:2026-07-24 15:08:05.13721765 +0000 UTC m=+0.356509288 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3230820 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a16757ea52fbd725c5615c5cb6f6 SystemUUID:ec27a167-57ea-52fb-d725-c5615c5cb6f6 BootID:ce506ac0-4e59-4be6-8276-8158e4c6a3ee Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e1:ec:54:33:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e1:ec:54:33:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:2c:dc:5a:32:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 15:08:05.138490 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.138484 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 15:08:05.138615 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.138575 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 15:08:05.139595 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.139565 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 15:08:05.139783 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.139598 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-115.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 15:08:05.140339 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.140329 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 15:08:05.140378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.140342 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 15:08:05.140378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.140350 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:08:05.140378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.140367 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:08:05.141226 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.141215 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 15:08:05.142951 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.142937 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:08:05.143066 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143056 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 15:08:05.143103 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143088 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 24 15:08:05.143103 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143099 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 15:08:05.143158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143110 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 15:08:05.143158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143121 2563 kubelet.go:398] "Adding apiserver pod source" Jul 24 15:08:05.143158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143130 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 15:08:05.143515 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143502 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:08:05.143595 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.143521 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:08:05.145642 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.145621 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 15:08:05.147793 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.147779 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 15:08:05.147861 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.147809 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 15:08:05.149338 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149325 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149343 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149349 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149354 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149363 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149369 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149377 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 15:08:05.149383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149383 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 15:08:05.149555 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149393 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 15:08:05.149555 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149402 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 15:08:05.149555 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149414 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 15:08:05.149555 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.149423 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 15:08:05.151002 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.150988 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 15:08:05.151002 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.151003 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 15:08:05.153531 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.153500 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-115.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 15:08:05.154102 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.154081 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 15:08:05.154172 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.154078 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 15:08:05.154717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.154691 2563 server.go:1268] "Started kubelet" Jul 24 15:08:05.154803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.154748 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 15:08:05.154803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.154758 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 15:08:05.154803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.154797 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 15:08:05.155424 ip-10-0-132-115 systemd[1]: Started Kubernetes Kubelet. Jul 24 15:08:05.156828 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.156813 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 24 15:08:05.157009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.156922 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 15:08:05.163739 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.163714 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 15:08:05.166016 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.165994 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 15:08:05.166111 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166016 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 15:08:05.166111 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.166072 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.166750 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.165848 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18c5424389348540 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:ip-10-0-132-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-07-24 15:08:05.15443232 +0000 UTC m=+0.373723980,LastTimestamp:2026-07-24 15:08:05.15443232 +0000 UTC m=+0.373723980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Jul 24 15:08:05.166864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166779 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 15:08:05.166864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166807 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 15:08:05.166864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166819 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 15:08:05.167020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166938 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 24 15:08:05.167020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166949 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 24 15:08:05.167020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.166987 2563 factory.go:55] Registering systemd factory Jul 24 15:08:05.167020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167004 2563 factory.go:223] Registration of the systemd container factory successfully Jul 24 15:08:05.167243 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167231 2563 factory.go:153] Registering CRI-O factory Jul 24 15:08:05.167310 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167247 2563 factory.go:223] Registration of the crio container factory successfully Jul 24 15:08:05.167362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167312 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 15:08:05.167362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167338 2563 factory.go:103] Registering Raw factory Jul 24 15:08:05.167362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167352 2563 manager.go:1196] Started watching for new ooms in manager Jul 24 15:08:05.167707 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.167694 2563 manager.go:349] Starting recovery of all containers Jul 24 15:08:05.176134 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.176099 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 15:08:05.176310 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.176283 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-115.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 15:08:05.177965 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.177948 2563 manager.go:354] Recovery completed Jul 24 15:08:05.182927 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.182914 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.184880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.184863 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.184954 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.184895 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.184954 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.184908 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.185478 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.185458 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 15:08:05.185478 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.185468 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.185478 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.185471 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 15:08:05.185647 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.185498 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:08:05.186198 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.186177 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.186262 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.186217 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.186262 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.186239 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.187204 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.187190 2563 policy_none.go:49] "None policy: Start" Jul 24 15:08:05.187279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.187210 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 15:08:05.187279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.187223 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 15:08:05.188235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.188220 2563 policy_none.go:47] "Start" Jul 24 15:08:05.195015 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.194941 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18c542438b051bfd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:ip-10-0-132-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-115.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-07-24 15:08:05.184879613 +0000 UTC m=+0.404171250,LastTimestamp:2026-07-24 15:08:05.184879613 +0000 UTC m=+0.404171250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Jul 24 15:08:05.198228 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.198207 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b52ts" Jul 24 15:08:05.204102 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.204074 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b52ts" Jul 24 15:08:05.204362 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.204283 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-115.ec2.internal.18c542438b0573b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-115.ec2.internal,UID:ip-10-0-132-115.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-115.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-115.ec2.internal,},FirstTimestamp:2026-07-24 15:08:05.184902065 +0000 UTC m=+0.404193701,LastTimestamp:2026-07-24 15:08:05.184902065 +0000 UTC m=+0.404193701,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-115.ec2.internal,}" Jul 24 15:08:05.216313 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216292 2563 manager.go:337] "Starting Device Plugin manager" Jul 24 15:08:05.216427 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.216325 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 15:08:05.216427 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216336 2563 server.go:87] "Starting device plugin registration server" Jul 24 15:08:05.216648 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216635 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 15:08:05.216693 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216650 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 15:08:05.216784 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216763 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 15:08:05.216861 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216845 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 15:08:05.216861 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.216857 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 15:08:05.217723 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.217481 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 15:08:05.217827 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.217766 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.233070 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.233045 2563 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2595 into the system container \"/system.slice\": " Jul 24 15:08:05.296079 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.296047 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 15:08:05.297376 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.297352 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 15:08:05.297462 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.297405 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 15:08:05.297462 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.297434 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 15:08:05.297551 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.297475 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 15:08:05.300745 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.300723 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:05.317323 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.317293 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.318448 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.318430 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.318581 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.318465 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.318581 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.318479 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.318581 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.318509 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.327200 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.327101 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.327200 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.327135 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-115.ec2.internal\": node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.358476 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.358442 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.398351 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.398309 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal"] Jul 24 15:08:05.398544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.398400 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.399956 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.399937 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.400053 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.399970 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.400053 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.399981 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.401259 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.401244 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.401397 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.401381 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.401451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.401414 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.402059 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.402037 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.402059 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.402052 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.402176 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.402074 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.402176 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.402088 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.402232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.402074 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.402232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.402222 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.403169 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.403154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.403231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.403184 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:05.403877 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.403858 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:05.403966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.403886 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:05.403966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.403898 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:05.429477 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.429447 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-115.ec2.internal\" not found" node="ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.434189 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.434170 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-115.ec2.internal\" not found" node="ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.459063 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.459027 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.469448 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.469418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4ace6b3f39731b3e1514dbf278e40d7-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"a4ace6b3f39731b3e1514dbf278e40d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.469550 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.469451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9978077abb98db8e9e58483f78a68846-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"9978077abb98db8e9e58483f78a68846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.469550 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.469476 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9978077abb98db8e9e58483f78a68846-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"9978077abb98db8e9e58483f78a68846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.559557 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.559515 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.569966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.569939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9978077abb98db8e9e58483f78a68846-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"9978077abb98db8e9e58483f78a68846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.570065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.569971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4ace6b3f39731b3e1514dbf278e40d7-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"a4ace6b3f39731b3e1514dbf278e40d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.570065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.569988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9978077abb98db8e9e58483f78a68846-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"9978077abb98db8e9e58483f78a68846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.570065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.570029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9978077abb98db8e9e58483f78a68846-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"9978077abb98db8e9e58483f78a68846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.570180 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.570061 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9978077abb98db8e9e58483f78a68846-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal\" (UID: \"9978077abb98db8e9e58483f78a68846\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.570180 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.570062 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4ace6b3f39731b3e1514dbf278e40d7-config\") pod \"kube-apiserver-proxy-ip-10-0-132-115.ec2.internal\" (UID: \"a4ace6b3f39731b3e1514dbf278e40d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.660504 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.660420 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.732023 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.731989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.738024 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:05.737996 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Jul 24 15:08:05.761255 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.761216 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.861711 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.861671 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:05.962234 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:05.962149 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:07.195525 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:07.195492 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:07.234981 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.234950 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 15:08:07.235482 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.235143 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:08:07.295966 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:07.295935 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:07.340947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.340900 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:03:06 +0000 UTC" deadline="2028-01-29 07:45:47.290041237 +0000 UTC" Jul 24 15:08:07.340947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.340934 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13288h37m39.949109905s" Jul 24 15:08:07.396292 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:07.396207 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-115.ec2.internal\" not found" Jul 24 15:08:07.428097 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.428071 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:07.499615 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.499592 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" Jul 24 15:08:07.515050 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.515019 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:07.519351 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.519330 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:08:07.520273 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.520255 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" Jul 24 15:08:07.529323 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.529296 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:08:07.541277 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:07.541229 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9978077abb98db8e9e58483f78a68846.slice/crio-1ace6ce91168dfe3ee0fd7a4fce4ce56df0be5d2a5c269ac90a151d692225057 WatchSource:0}: Error finding container 1ace6ce91168dfe3ee0fd7a4fce4ce56df0be5d2a5c269ac90a151d692225057: Status 404 returned error can't find the container with id 1ace6ce91168dfe3ee0fd7a4fce4ce56df0be5d2a5c269ac90a151d692225057 Jul 24 15:08:07.543933 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:07.543906 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ace6b3f39731b3e1514dbf278e40d7.slice/crio-36a0bc9ad0c6b22b28248e13b6320b9ab9aa61c5f7556c4d65dc6fcc30badad6 WatchSource:0}: Error finding container 36a0bc9ad0c6b22b28248e13b6320b9ab9aa61c5f7556c4d65dc6fcc30badad6: Status 404 returned error can't find the container with id 36a0bc9ad0c6b22b28248e13b6320b9ab9aa61c5f7556c4d65dc6fcc30badad6 Jul 24 15:08:07.547698 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.547677 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:08:07.711222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:07.711141 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:08.161094 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.161003 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:08.277213 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.277171 2563 apiserver.go:52] "Watching apiserver" Jul 24 15:08:08.284826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.284791 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 15:08:08.285410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.285376 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-656vw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal","openshift-multus/multus-additional-cni-plugins-ccj4t","openshift-multus/multus-zcrc8","openshift-network-operator/iptables-alerter-5pcdg","kube-system/global-pull-secret-syncer-wb6w7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs","openshift-cluster-node-tuning-operator/tuned-nf9vk","openshift-image-registry/node-ca-g9bzd","openshift-multus/network-metrics-daemon-hsx6n","openshift-network-diagnostics/network-check-target-5h4x9","openshift-ovn-kubernetes/ovnkube-node-8xg7m","kube-system/konnectivity-agent-bdctf","kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal"] Jul 24 15:08:08.287361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.287093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.288457 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.288432 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.289802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.289778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.290488 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.290280 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 15:08:08.290488 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.290365 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.290953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.290762 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.290953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.290796 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74l4r\"" Jul 24 15:08:08.291099 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.291077 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.292556 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.292082 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.296606 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.296584 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 15:08:08.296830 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.296812 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bvn5n\"" Jul 24 15:08:08.297004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.296988 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 15:08:08.297315 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.297291 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 15:08:08.297522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.297505 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 15:08:08.297696 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.297679 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.298183 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298154 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 15:08:08.298183 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298185 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.298429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298392 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.298429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298404 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 15:08:08.298542 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298456 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.298542 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298478 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.298542 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298401 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.298542 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.298523 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:08.298680 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.298557 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:08.298680 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298659 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.298761 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298746 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.298866 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.298851 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m5zkn\"" Jul 24 15:08:08.299136 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.299120 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.299334 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.299314 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pmcj4\"" Jul 24 15:08:08.299677 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.299659 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 15:08:08.299812 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.299795 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4kmn\"" Jul 24 15:08:08.300029 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.300014 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 15:08:08.300304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.300286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:08.300399 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.300376 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:08.300999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.300939 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.301201 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301185 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4kpwh\"" Jul 24 15:08:08.301473 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301455 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.301541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301529 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.301672 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301657 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.301754 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301733 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.301809 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301795 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.301895 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.301879 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 15:08:08.302927 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.302908 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-485hn\"" Jul 24 15:08:08.303311 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.303282 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ffhpv\"" Jul 24 15:08:08.306290 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.306149 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 15:08:08.306542 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.306522 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-85mrc\"" Jul 24 15:08:08.307930 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.307904 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.308625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.308018 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 15:08:08.308625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.308261 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 15:08:08.308625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.308207 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 15:08:08.308625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.308537 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 15:08:08.310533 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.309069 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 15:08:08.311090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.310941 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-netns\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.311090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.310982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmtb5\" (UniqueName: \"kubernetes.io/projected/cd8df5f7-9834-411d-a67a-99a1a3394206-kube-api-access-zmtb5\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.311090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-sys\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.311090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311053 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-device-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/393ec1dc-8487-43e5-953d-969a2b7f10aa-agent-certs\") pod \"konnectivity-agent-bdctf\" (UID: \"393ec1dc-8487-43e5-953d-969a2b7f10aa\") " pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311284 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-cni-multus\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-os-release\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311428 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-hostroot\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.311459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-kubelet-config\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311508 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/367c5380-8129-47f1-a47a-7109462a8049-etc-tuned\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-socket-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-host\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-cnibin\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45e798c4-7d80-4bfc-8918-b0ad78e0d087-cni-binary-copy\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311721 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b956246-0399-4cb5-9375-532d4f544fc6-iptables-alerter-script\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.311799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-registration-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.312158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysctl-d\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.312158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-dbus\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.312158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.311972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.312158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysconfig\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.312158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-system-cni-dir\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-kubelet\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312186 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-conf-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b956246-0399-4cb5-9375-532d4f544fc6-host-slash\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312300 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhc7\" (UniqueName: \"kubernetes.io/projected/3d99c649-459a-407b-aea5-83b0dd0b1120-kube-api-access-dfhc7\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312338 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcjb\" (UniqueName: \"kubernetes.io/projected/84271708-62ea-4275-a46c-b8d088f96578-kube-api-access-mfcjb\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.312419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-multus-certs\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.312723 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.312449 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtk85\" (UniqueName: \"kubernetes.io/projected/45e798c4-7d80-4bfc-8918-b0ad78e0d087-kube-api-access-qtk85\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.313620 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.313364 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315775 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40253b47-eaff-437d-a540-2526e5e4f40d-hosts-file\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-sys-fs\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315851 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315874 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40253b47-eaff-437d-a540-2526e5e4f40d-tmp-dir\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-run\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315915 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-cni-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315936 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-cni-bin\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315959 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-etc-kubernetes\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.315981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-var-lib-kubelet\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.316002 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.316024 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/393ec1dc-8487-43e5-953d-969a2b7f10aa-konnectivity-ca\") pod \"konnectivity-agent-bdctf\" (UID: \"393ec1dc-8487-43e5-953d-969a2b7f10aa\") " pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.316047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-cni-binary-copy\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.316069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-system-cni-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.316135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.316090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-modprobe-d\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.316966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-lib-modules\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317015 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfvmq\" (UniqueName: \"kubernetes.io/projected/9b956246-0399-4cb5-9375-532d4f544fc6-kube-api-access-lfvmq\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317041 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kzf\" (UniqueName: \"kubernetes.io/projected/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-kube-api-access-w5kzf\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317068 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn569\" (UniqueName: \"kubernetes.io/projected/40253b47-eaff-437d-a540-2526e5e4f40d-kube-api-access-qn569\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-kubernetes\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysctl-conf\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-socket-dir-parent\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-systemd\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317183 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367c5380-8129-47f1-a47a-7109462a8049-tmp\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317205 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-host\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317227 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-etc-selinux\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-cnibin\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317286 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-os-release\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-daemon-config\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xzc\" (UniqueName: \"kubernetes.io/projected/367c5380-8129-47f1-a47a-7109462a8049-kube-api-access-h6xzc\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.318197 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-serviceca\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.319101 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.317384 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-k8s-cni-cncf-io\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.344605 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.344531 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fpm9t" Jul 24 15:08:08.360706 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.360677 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fpm9t" Jul 24 15:08:08.400353 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.400324 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.417979 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-log-socket\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5q77\" (UniqueName: \"kubernetes.io/projected/b543d8cd-803e-4799-9c92-46c820f7dae2-kube-api-access-m5q77\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-registration-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysctl-d\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-var-lib-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-cni-bin\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418151 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-ovnkube-script-lib\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-dbus\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418200 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysconfig\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-systemd-units\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418291 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-system-cni-dir\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-kubelet\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418336 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-conf-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b956246-0399-4cb5-9375-532d4f544fc6-host-slash\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhc7\" (UniqueName: \"kubernetes.io/projected/3d99c649-459a-407b-aea5-83b0dd0b1120-kube-api-access-dfhc7\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.418574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418408 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418414 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-registration-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcjb\" (UniqueName: \"kubernetes.io/projected/84271708-62ea-4275-a46c-b8d088f96578-kube-api-access-mfcjb\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418527 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418534 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-multus-certs\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtk85\" (UniqueName: \"kubernetes.io/projected/45e798c4-7d80-4bfc-8918-b0ad78e0d087-kube-api-access-qtk85\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-dbus\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.418815 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40253b47-eaff-437d-a540-2526e5e4f40d-hosts-file\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-sys-fs\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.418904 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:08.918867452 +0000 UTC m=+3.005519350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418915 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-sys-fs\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.418956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40253b47-eaff-437d-a540-2526e5e4f40d-tmp-dir\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-kubelet\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419033 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysctl-d\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.419522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-conf-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419080 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40253b47-eaff-437d-a540-2526e5e4f40d-hosts-file\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419092 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b956246-0399-4cb5-9375-532d4f544fc6-host-slash\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419213 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-multus-certs\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-run\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-kubelet\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-run-netns\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419344 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-system-cni-dir\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419346 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-etc-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-cni-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-cni-bin\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysconfig\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419415 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-etc-kubernetes\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-var-lib-kubelet\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419478 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-run\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419508 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-cni-bin\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419525 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/393ec1dc-8487-43e5-953d-969a2b7f10aa-konnectivity-ca\") pod \"konnectivity-agent-bdctf\" (UID: \"393ec1dc-8487-43e5-953d-969a2b7f10aa\") " pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.420297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419530 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-etc-kubernetes\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-cni-binary-copy\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419561 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-var-lib-kubelet\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-system-cni-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419597 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-modprobe-d\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419618 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-cni-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-lib-modules\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419706 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-system-cni-dir\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419711 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40253b47-eaff-437d-a540-2526e5e4f40d-tmp-dir\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419813 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-modprobe-d\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.419920 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.419948 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-cni-netd\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.420049 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:08.920031269 +0000 UTC m=+3.006683172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-env-overrides\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420122 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b543d8cd-803e-4799-9c92-46c820f7dae2-ovn-node-metrics-cert\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420138 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-cni-binary-copy\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.421065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420154 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-lib-modules\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfvmq\" (UniqueName: \"kubernetes.io/projected/9b956246-0399-4cb5-9375-532d4f544fc6-kube-api-access-lfvmq\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420198 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kzf\" (UniqueName: \"kubernetes.io/projected/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-kube-api-access-w5kzf\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn569\" (UniqueName: \"kubernetes.io/projected/40253b47-eaff-437d-a540-2526e5e4f40d-kube-api-access-qn569\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-kubernetes\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysctl-conf\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-socket-dir-parent\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-systemd\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420393 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-kubernetes\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420405 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/393ec1dc-8487-43e5-953d-969a2b7f10aa-konnectivity-ca\") pod \"konnectivity-agent-bdctf\" (UID: \"393ec1dc-8487-43e5-953d-969a2b7f10aa\") " pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420395 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-systemd\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420517 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-etc-sysctl-conf\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-socket-dir-parent\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367c5380-8129-47f1-a47a-7109462a8049-tmp\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420613 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-systemd\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-node-log\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.421933 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-host\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420754 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-etc-selinux\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420788 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-host\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-cnibin\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420849 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-os-release\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420871 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-cnibin\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420928 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-daemon-config\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420939 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420963 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-os-release\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.420982 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xzc\" (UniqueName: \"kubernetes.io/projected/367c5380-8129-47f1-a47a-7109462a8049-kube-api-access-h6xzc\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-etc-selinux\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-ovn\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-serviceca\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-k8s-cni-cncf-io\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-netns\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmtb5\" (UniqueName: \"kubernetes.io/projected/cd8df5f7-9834-411d-a67a-99a1a3394206-kube-api-access-zmtb5\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-sys\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.422756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421336 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-device-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-k8s-cni-cncf-io\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/393ec1dc-8487-43e5-953d-969a2b7f10aa-agent-certs\") pod \"konnectivity-agent-bdctf\" (UID: \"393ec1dc-8487-43e5-953d-969a2b7f10aa\") " pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-cni-multus\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-os-release\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421479 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-hostroot\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-kubelet-config\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421510 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45e798c4-7d80-4bfc-8918-b0ad78e0d087-multus-daemon-config\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421527 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/367c5380-8129-47f1-a47a-7109462a8049-etc-tuned\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421595 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-sys\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421640 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-run-netns\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421653 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-serviceca\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421846 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-hostroot\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421905 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-os-release\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/84271708-62ea-4275-a46c-b8d088f96578-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.423541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421969 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-device-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.421974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-ovnkube-config\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-kubelet-config\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422063 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-host-var-lib-cni-multus\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-socket-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422135 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-host\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-cnibin\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45e798c4-7d80-4bfc-8918-b0ad78e0d087-cni-binary-copy\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422202 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422220 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b956246-0399-4cb5-9375-532d4f544fc6-iptables-alerter-script\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422227 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/367c5380-8129-47f1-a47a-7109462a8049-host\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422268 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-slash\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45e798c4-7d80-4bfc-8918-b0ad78e0d087-cnibin\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422357 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cd8df5f7-9834-411d-a67a-99a1a3394206-socket-dir\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422533 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84271708-62ea-4275-a46c-b8d088f96578-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.422822 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45e798c4-7d80-4bfc-8918-b0ad78e0d087-cni-binary-copy\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.424466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.423062 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b956246-0399-4cb5-9375-532d4f544fc6-iptables-alerter-script\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.428096 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.428024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/367c5380-8129-47f1-a47a-7109462a8049-tmp\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.429810 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.429354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtk85\" (UniqueName: \"kubernetes.io/projected/45e798c4-7d80-4bfc-8918-b0ad78e0d087-kube-api-access-qtk85\") pod \"multus-zcrc8\" (UID: \"45e798c4-7d80-4bfc-8918-b0ad78e0d087\") " pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.429810 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.429746 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/393ec1dc-8487-43e5-953d-969a2b7f10aa-agent-certs\") pod \"konnectivity-agent-bdctf\" (UID: \"393ec1dc-8487-43e5-953d-969a2b7f10aa\") " pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.434141 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.430117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/367c5380-8129-47f1-a47a-7109462a8049-etc-tuned\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.434141 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.432227 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn569\" (UniqueName: \"kubernetes.io/projected/40253b47-eaff-437d-a540-2526e5e4f40d-kube-api-access-qn569\") pod \"node-resolver-656vw\" (UID: \"40253b47-eaff-437d-a540-2526e5e4f40d\") " pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.434141 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.433322 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhc7\" (UniqueName: \"kubernetes.io/projected/3d99c649-459a-407b-aea5-83b0dd0b1120-kube-api-access-dfhc7\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.434400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.434224 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcjb\" (UniqueName: \"kubernetes.io/projected/84271708-62ea-4275-a46c-b8d088f96578-kube-api-access-mfcjb\") pod \"multus-additional-cni-plugins-ccj4t\" (UID: \"84271708-62ea-4275-a46c-b8d088f96578\") " pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.435155 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.434650 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmtb5\" (UniqueName: \"kubernetes.io/projected/cd8df5f7-9834-411d-a67a-99a1a3394206-kube-api-access-zmtb5\") pod \"aws-ebs-csi-driver-node-dttgs\" (UID: \"cd8df5f7-9834-411d-a67a-99a1a3394206\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.435155 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.434923 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:08.435155 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.434941 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:08.435155 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.434954 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:08.435155 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.435012 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:08.934993305 +0000 UTC m=+3.021645212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:08.437845 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.437786 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfvmq\" (UniqueName: \"kubernetes.io/projected/9b956246-0399-4cb5-9375-532d4f544fc6-kube-api-access-lfvmq\") pod \"iptables-alerter-5pcdg\" (UID: \"9b956246-0399-4cb5-9375-532d4f544fc6\") " pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.439638 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.439585 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" event={"ID":"a4ace6b3f39731b3e1514dbf278e40d7","Type":"ContainerStarted","Data":"36a0bc9ad0c6b22b28248e13b6320b9ab9aa61c5f7556c4d65dc6fcc30badad6"} Jul 24 15:08:08.440451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.440282 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xzc\" (UniqueName: \"kubernetes.io/projected/367c5380-8129-47f1-a47a-7109462a8049-kube-api-access-h6xzc\") pod \"tuned-nf9vk\" (UID: \"367c5380-8129-47f1-a47a-7109462a8049\") " pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.440556 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.440464 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kzf\" (UniqueName: \"kubernetes.io/projected/d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0-kube-api-access-w5kzf\") pod \"node-ca-g9bzd\" (UID: \"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0\") " pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.441409 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.441381 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"9978077abb98db8e9e58483f78a68846","Type":"ContainerStarted","Data":"1ace6ce91168dfe3ee0fd7a4fce4ce56df0be5d2a5c269ac90a151d692225057"} Jul 24 15:08:08.523250 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523250 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-ovnkube-config\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-slash\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-log-socket\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523333 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523343 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5q77\" (UniqueName: \"kubernetes.io/projected/b543d8cd-803e-4799-9c92-46c820f7dae2-kube-api-access-m5q77\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-var-lib-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523429 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-cni-bin\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-ovnkube-script-lib\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-systemd-units\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-kubelet\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-run-netns\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523600 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-etc-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523622 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-slash\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-cni-netd\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523668 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-cni-bin\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-env-overrides\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523691 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-cni-netd\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523712 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-kubelet\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523715 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b543d8cd-803e-4799-9c92-46c820f7dae2-ovn-node-metrics-cert\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523737 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-run-netns\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-systemd\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-systemd-units\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-node-log\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523779 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-etc-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.523849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523790 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523834 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-ovn\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523872 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-ovn\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523631 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-var-lib-openvswitch\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-ovnkube-config\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523916 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-node-log\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.523925 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-run-systemd\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.524091 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b543d8cd-803e-4799-9c92-46c820f7dae2-log-socket\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.524220 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-ovnkube-script-lib\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.524661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.524359 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b543d8cd-803e-4799-9c92-46c820f7dae2-env-overrides\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.526423 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.526392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b543d8cd-803e-4799-9c92-46c820f7dae2-ovn-node-metrics-cert\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.531934 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.531908 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5q77\" (UniqueName: \"kubernetes.io/projected/b543d8cd-803e-4799-9c92-46c820f7dae2-kube-api-access-m5q77\") pod \"ovnkube-node-8xg7m\" (UID: \"b543d8cd-803e-4799-9c92-46c820f7dae2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.610925 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.610887 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" Jul 24 15:08:08.621445 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.620717 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:08.621445 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:08.621152 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8df5f7_9834_411d_a67a_99a1a3394206.slice/crio-50eef7760fe053271f83250179761cd135dfaa02c9b8cb9ebdf759a0f7d2f5b0 WatchSource:0}: Error finding container 50eef7760fe053271f83250179761cd135dfaa02c9b8cb9ebdf759a0f7d2f5b0: Status 404 returned error can't find the container with id 50eef7760fe053271f83250179761cd135dfaa02c9b8cb9ebdf759a0f7d2f5b0 Jul 24 15:08:08.629909 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:08.629881 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393ec1dc_8487_43e5_953d_969a2b7f10aa.slice/crio-e0eb91c2f89360ca42c81ce3836669226e18f04e2c67fa8cf5b7d9600a7a3b51 WatchSource:0}: Error finding container e0eb91c2f89360ca42c81ce3836669226e18f04e2c67fa8cf5b7d9600a7a3b51: Status 404 returned error can't find the container with id e0eb91c2f89360ca42c81ce3836669226e18f04e2c67fa8cf5b7d9600a7a3b51 Jul 24 15:08:08.631528 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.631505 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" Jul 24 15:08:08.636999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.636974 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zcrc8" Jul 24 15:08:08.645722 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.645694 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pcdg" Jul 24 15:08:08.653574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.653546 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" Jul 24 15:08:08.662287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.662263 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9bzd" Jul 24 15:08:08.671928 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.671858 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-656vw" Jul 24 15:08:08.679039 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.679010 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:08.926374 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.926282 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:08.926374 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:08.926346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:08.926586 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.926459 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:08.926586 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.926462 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:08.926586 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.926512 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:09.926496818 +0000 UTC m=+4.013148710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:08.926586 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:08.926525 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:09.926519055 +0000 UTC m=+4.013170946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:09.027681 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.027648 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:09.027885 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.027827 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:09.027885 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.027852 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:09.027885 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.027865 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:09.028044 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.027932 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:10.027911754 +0000 UTC m=+4.114563649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:09.363069 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.362973 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:03:08 +0000 UTC" deadline="2028-03-24 10:30:33.755185161 +0000 UTC" Jul 24 15:08:09.363069 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.363010 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14611h22m24.392178417s" Jul 24 15:08:09.430656 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.430621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:09.430817 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.430748 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:09.445110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.445061 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdctf" event={"ID":"393ec1dc-8487-43e5-953d-969a2b7f10aa","Type":"ContainerStarted","Data":"e0eb91c2f89360ca42c81ce3836669226e18f04e2c67fa8cf5b7d9600a7a3b51"} Jul 24 15:08:09.446667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.446634 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" event={"ID":"cd8df5f7-9834-411d-a67a-99a1a3394206","Type":"ContainerStarted","Data":"50eef7760fe053271f83250179761cd135dfaa02c9b8cb9ebdf759a0f7d2f5b0"} Jul 24 15:08:09.599745 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.599688 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84271708_62ea_4275_a46c_b8d088f96578.slice/crio-9fa5ccc0aa86509a1ba9039ed3f3ec0233239d98ad435364713a48e3758172f2 WatchSource:0}: Error finding container 9fa5ccc0aa86509a1ba9039ed3f3ec0233239d98ad435364713a48e3758172f2: Status 404 returned error can't find the container with id 9fa5ccc0aa86509a1ba9039ed3f3ec0233239d98ad435364713a48e3758172f2 Jul 24 15:08:09.628005 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.627976 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367c5380_8129_47f1_a47a_7109462a8049.slice/crio-dde8dfbb2ad217ae3383534f17ed3ffc2e88e57d7c027b965eb47a3b02a8e588 WatchSource:0}: Error finding container dde8dfbb2ad217ae3383534f17ed3ffc2e88e57d7c027b965eb47a3b02a8e588: Status 404 returned error can't find the container with id dde8dfbb2ad217ae3383534f17ed3ffc2e88e57d7c027b965eb47a3b02a8e588 Jul 24 15:08:09.628748 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.628723 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40253b47_eaff_437d_a540_2526e5e4f40d.slice/crio-08c4f001a3bacbce8522a92c450b97917a21f25ea6938479b18c54cee396e26c WatchSource:0}: Error finding container 08c4f001a3bacbce8522a92c450b97917a21f25ea6938479b18c54cee396e26c: Status 404 returned error can't find the container with id 08c4f001a3bacbce8522a92c450b97917a21f25ea6938479b18c54cee396e26c Jul 24 15:08:09.630674 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.630297 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dfc302_e6b9_43f0_b89f_b264a4d5a2a0.slice/crio-a38af43cb14aba1ed6cc6df94cfb4cf283084bd18b010d67e24d6409a3f98fc0 WatchSource:0}: Error finding container a38af43cb14aba1ed6cc6df94cfb4cf283084bd18b010d67e24d6409a3f98fc0: Status 404 returned error can't find the container with id a38af43cb14aba1ed6cc6df94cfb4cf283084bd18b010d67e24d6409a3f98fc0 Jul 24 15:08:09.631812 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.631768 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e798c4_7d80_4bfc_8918_b0ad78e0d087.slice/crio-dccf58e8959a134d0b27d0ed753cff9674c54987146f59284737750895618ebf WatchSource:0}: Error finding container dccf58e8959a134d0b27d0ed753cff9674c54987146f59284737750895618ebf: Status 404 returned error can't find the container with id dccf58e8959a134d0b27d0ed753cff9674c54987146f59284737750895618ebf Jul 24 15:08:09.633436 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.633394 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb543d8cd_803e_4799_9c92_46c820f7dae2.slice/crio-4123fab42bd75144f374b81f3aca2c62fef2c2ff880082a5f85df7948d388dd2 WatchSource:0}: Error finding container 4123fab42bd75144f374b81f3aca2c62fef2c2ff880082a5f85df7948d388dd2: Status 404 returned error can't find the container with id 4123fab42bd75144f374b81f3aca2c62fef2c2ff880082a5f85df7948d388dd2 Jul 24 15:08:09.634755 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:09.634731 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b956246_0399_4cb5_9375_532d4f544fc6.slice/crio-73c68f25e9e3ebdbc04b37d1d9fa32a0fe1ee4d0871ce348cc62f6a0bd056907 WatchSource:0}: Error finding container 73c68f25e9e3ebdbc04b37d1d9fa32a0fe1ee4d0871ce348cc62f6a0bd056907: Status 404 returned error can't find the container with id 73c68f25e9e3ebdbc04b37d1d9fa32a0fe1ee4d0871ce348cc62f6a0bd056907 Jul 24 15:08:09.933450 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.933210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:09.933602 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:09.933486 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:09.933602 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.933356 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:09.933602 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.933578 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:11.933559352 +0000 UTC m=+6.020211260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:09.933602 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.933601 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:09.933729 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:09.933634 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:11.933623724 +0000 UTC m=+6.020275616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:10.034077 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.034029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:10.034479 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:10.034347 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:10.034479 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:10.034376 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:10.034479 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:10.034390 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:10.034479 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:10.034456 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:12.034438309 +0000 UTC m=+6.121090207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:10.363361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.363179 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:03:08 +0000 UTC" deadline="2027-12-24 12:46:36.783027093 +0000 UTC" Jul 24 15:08:10.363361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.363234 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12429h38m26.419797412s" Jul 24 15:08:10.434303 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.433635 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:10.434303 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:10.433772 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:10.434303 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.434145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:10.434303 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:10.434230 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:10.466908 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.466827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" event={"ID":"367c5380-8129-47f1-a47a-7109462a8049","Type":"ContainerStarted","Data":"dde8dfbb2ad217ae3383534f17ed3ffc2e88e57d7c027b965eb47a3b02a8e588"} Jul 24 15:08:10.485524 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.482138 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9bzd" event={"ID":"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0","Type":"ContainerStarted","Data":"a38af43cb14aba1ed6cc6df94cfb4cf283084bd18b010d67e24d6409a3f98fc0"} Jul 24 15:08:10.494772 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.494453 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerStarted","Data":"9fa5ccc0aa86509a1ba9039ed3f3ec0233239d98ad435364713a48e3758172f2"} Jul 24 15:08:10.499720 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.499679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" event={"ID":"a4ace6b3f39731b3e1514dbf278e40d7","Type":"ContainerStarted","Data":"75b8ee0d2153e56dfb5314c2243bfd661cda30d2c029cfd603045d4f1001be01"} Jul 24 15:08:10.513407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.512719 2563 generic.go:356] "Generic (PLEG): container finished" podID="9978077abb98db8e9e58483f78a68846" containerID="f214fd07e75229e1f05a1fb62fa6e7e44bd7c9e4b1477e1c4ec6115cce9709c0" exitCode=0 Jul 24 15:08:10.513407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.512815 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"9978077abb98db8e9e58483f78a68846","Type":"ContainerDied","Data":"f214fd07e75229e1f05a1fb62fa6e7e44bd7c9e4b1477e1c4ec6115cce9709c0"} Jul 24 15:08:10.517629 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.517512 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pcdg" event={"ID":"9b956246-0399-4cb5-9375-532d4f544fc6","Type":"ContainerStarted","Data":"73c68f25e9e3ebdbc04b37d1d9fa32a0fe1ee4d0871ce348cc62f6a0bd056907"} Jul 24 15:08:10.519436 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.519406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zcrc8" event={"ID":"45e798c4-7d80-4bfc-8918-b0ad78e0d087","Type":"ContainerStarted","Data":"dccf58e8959a134d0b27d0ed753cff9674c54987146f59284737750895618ebf"} Jul 24 15:08:10.521997 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.521942 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-115.ec2.internal" podStartSLOduration=3.521923945 podStartE2EDuration="3.521923945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:10.520816314 +0000 UTC m=+4.607468228" watchObservedRunningTime="2026-07-24 15:08:10.521923945 +0000 UTC m=+4.608575859" Jul 24 15:08:10.531069 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.531030 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"4123fab42bd75144f374b81f3aca2c62fef2c2ff880082a5f85df7948d388dd2"} Jul 24 15:08:10.533632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:10.533602 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-656vw" event={"ID":"40253b47-eaff-437d-a540-2526e5e4f40d","Type":"ContainerStarted","Data":"08c4f001a3bacbce8522a92c450b97917a21f25ea6938479b18c54cee396e26c"} Jul 24 15:08:11.430852 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:11.430772 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:11.431350 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:11.430903 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:11.952378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:11.951682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:11.952378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:11.951737 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:11.952378 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:11.951867 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:11.952378 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:11.951930 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:15.951911915 +0000 UTC m=+10.038563811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:11.952378 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:11.952013 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:11.952378 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:11.952053 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:15.952041349 +0000 UTC m=+10.038693244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:12.053570 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:12.052917 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:12.053570 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:12.053155 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:12.053570 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:12.053175 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:12.053570 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:12.053187 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:12.053570 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:12.053256 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:16.053223632 +0000 UTC m=+10.139875531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:12.432205 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:12.431366 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:12.432205 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:12.431522 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:12.432205 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:12.431981 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:12.432205 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:12.432074 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:13.430893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:13.430856 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:13.431063 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:13.431000 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:14.432463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:14.430492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:14.432463 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:14.430653 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:14.432463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:14.431128 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:14.432463 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:14.431219 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:15.431256 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:15.431154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:15.431415 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:15.431309 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:15.987523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:15.987300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:15.987523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:15.987398 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:15.987523 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:15.987453 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:15.987523 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:15.987482 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:15.987523 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:15.987521 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:23.987504072 +0000 UTC m=+18.074155967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:15.988194 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:15.987537 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:23.98752931 +0000 UTC m=+18.074181201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:16.088761 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:16.088673 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:16.088925 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:16.088856 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:16.088925 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:16.088883 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:16.088925 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:16.088896 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:16.089056 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:16.088965 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:24.088945309 +0000 UTC m=+18.175597204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:16.434505 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:16.433983 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:16.434505 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:16.434099 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:16.434505 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:16.434308 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:16.434505 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:16.434413 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:17.430710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:17.430666 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:17.431228 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:17.430819 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:18.430350 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:18.430312 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:18.430518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:18.430322 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:18.430518 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:18.430465 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:18.430518 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:18.430498 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:19.430655 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:19.430569 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:19.431040 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:19.430699 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:20.430509 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:20.430473 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:20.430709 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:20.430475 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:20.430709 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:20.430627 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:20.430709 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:20.430677 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:21.430407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:21.430376 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:21.430570 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:21.430488 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:22.431162 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:22.431131 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:22.431633 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:22.431282 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:22.431633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:22.431344 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:22.431633 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:22.431464 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:23.430294 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:23.430257 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:23.430515 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:23.430383 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:24.044294 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:24.044235 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:24.044742 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:24.044315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:24.044742 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.044431 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:24.044742 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.044463 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:24.044742 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.044516 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:40.044493965 +0000 UTC m=+34.131145880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:24.044742 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.044540 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:40.044529391 +0000 UTC m=+34.131181282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:24.145485 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:24.145434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:24.145655 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.145617 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:24.145655 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.145637 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:24.145655 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.145650 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:24.145796 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.145706 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:40.145691564 +0000 UTC m=+34.232343455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:24.430870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:24.430783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:24.431039 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:24.430783 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:24.431039 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.430935 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:24.431039 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:24.430997 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:25.431289 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:25.431233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:25.431792 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:25.431386 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:26.431479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:26.431447 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:26.431935 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:26.431567 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:26.431935 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:26.431630 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:26.431935 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:26.431740 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:27.430745 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.430582 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:27.430876 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:27.430850 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:27.574218 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.574141 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zcrc8" event={"ID":"45e798c4-7d80-4bfc-8918-b0ad78e0d087","Type":"ContainerStarted","Data":"a5538d46f722f5e0f581dddf2a3f2195be2ff49431ab8b52418cc945565a5583"} Jul 24 15:08:27.576138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.575878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-656vw" event={"ID":"40253b47-eaff-437d-a540-2526e5e4f40d","Type":"ContainerStarted","Data":"dedb1c870c2bc3b03bc9729cf6fc609b9d5b68ec120b7696fc77d20b2da510ed"} Jul 24 15:08:27.577739 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.577707 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" event={"ID":"367c5380-8129-47f1-a47a-7109462a8049","Type":"ContainerStarted","Data":"489ff20883afec39a31d8295eeef0141a5cf1c55548259281c5e909ec50a08aa"} Jul 24 15:08:27.579380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.579353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bdctf" event={"ID":"393ec1dc-8487-43e5-953d-969a2b7f10aa","Type":"ContainerStarted","Data":"428784ff346689c7affe55cdc6b1dba68d0ab5678be215b0b229e4d93b4d9b07"} Jul 24 15:08:27.581114 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.581088 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9bzd" event={"ID":"d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0","Type":"ContainerStarted","Data":"f128cffa16808627509aa5f406a529e014edb7a30662362677bb838204ed29c7"} Jul 24 15:08:27.582861 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.582838 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerStarted","Data":"d42e98c9a48e8ca12be3d481a6b637ae84777718a42893b58930e41489a96420"} Jul 24 15:08:27.585134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.585106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" event={"ID":"9978077abb98db8e9e58483f78a68846","Type":"ContainerStarted","Data":"4bd90b1c4718bcfa62e63b255d0bdd109c61e35a3d4ac4246ae3ad20dadcf391"} Jul 24 15:08:27.586725 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.586688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" event={"ID":"cd8df5f7-9834-411d-a67a-99a1a3394206","Type":"ContainerStarted","Data":"4991808376f262115c7ce51485d176c08cefa9ca4bd912a942279dfcbf4c6cec"} Jul 24 15:08:27.599179 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.599121 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zcrc8" podStartSLOduration=4.011397933 podStartE2EDuration="21.599097173s" totalImagesPullingTime="17.58769924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:27.597970043 +0000 UTC m=+21.684621957" watchObservedRunningTime="2026-07-24 15:08:27.599097173 +0000 UTC m=+21.685749088" Jul 24 15:08:27.616710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.616660 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g9bzd" podStartSLOduration=4.081985273 podStartE2EDuration="21.616642205s" totalImagesPullingTime="17.534656932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:27.616073806 +0000 UTC m=+21.702725718" watchObservedRunningTime="2026-07-24 15:08:27.616642205 +0000 UTC m=+21.703294117" Jul 24 15:08:27.633993 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.633941 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-656vw" podStartSLOduration=4.077182383 podStartE2EDuration="21.633923316s" totalImagesPullingTime="17.556740933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:27.633841285 +0000 UTC m=+21.720493198" watchObservedRunningTime="2026-07-24 15:08:27.633923316 +0000 UTC m=+21.720575230" Jul 24 15:08:27.651306 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.651235 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bdctf" podStartSLOduration=3.096129087 podStartE2EDuration="21.651219256s" totalImagesPullingTime="18.555090169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:27.650958251 +0000 UTC m=+21.737610164" watchObservedRunningTime="2026-07-24 15:08:27.651219256 +0000 UTC m=+21.737871188" Jul 24 15:08:27.730797 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.730751 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nf9vk" podStartSLOduration=4.139883027 podStartE2EDuration="21.73073498s" totalImagesPullingTime="17.590851953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:27.706861499 +0000 UTC m=+21.793513413" watchObservedRunningTime="2026-07-24 15:08:27.73073498 +0000 UTC m=+21.817386908" Jul 24 15:08:27.731193 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:27.731164 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-115.ec2.internal" podStartSLOduration=20.731156202 podStartE2EDuration="20.731156202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:27.730564295 +0000 UTC m=+21.817216203" watchObservedRunningTime="2026-07-24 15:08:27.731156202 +0000 UTC m=+21.817808116" Jul 24 15:08:28.430922 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.430732 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:28.431101 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.430762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:28.431101 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:28.431025 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:28.431101 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:28.431054 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:28.592017 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.591995 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592330 2563 generic.go:356] "Generic (PLEG): container finished" podID="b543d8cd-803e-4799-9c92-46c820f7dae2" containerID="da55bbe29660ce8d4ae69315a0603e0a2529945ae24c80b5cea46f3b51fb4448" exitCode=1 Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"aecae708a4fe378d54ff437c1986e0e505495dcec71e833250fc034c535fc880"} Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592470 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"eb084772f1a1a869efac9d3f5995e3727e4b22e044f99de85311eb36b251ad33"} Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"7307bf016c846cef4c63de71e14e44ede138c0c0f79a265ca7e9b0a2ff5b5dc1"} Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592528 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"f75910f430ff07651fc51de78e556a748bc1940dace5abcb792f17af0f54050d"} Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592554 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerDied","Data":"da55bbe29660ce8d4ae69315a0603e0a2529945ae24c80b5cea46f3b51fb4448"} Jul 24 15:08:28.592665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.592597 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"c1eece984cc47be5457abf03fc4326140c128199932b059980a34f36c3974ad1"} Jul 24 15:08:28.593803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.593780 2563 generic.go:356] "Generic (PLEG): container finished" podID="84271708-62ea-4275-a46c-b8d088f96578" containerID="d42e98c9a48e8ca12be3d481a6b637ae84777718a42893b58930e41489a96420" exitCode=0 Jul 24 15:08:28.593913 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.593859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerDied","Data":"d42e98c9a48e8ca12be3d481a6b637ae84777718a42893b58930e41489a96420"} Jul 24 15:08:28.612775 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:28.612753 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 15:08:29.361304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.361187 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:08:28.612768374Z","UUID":"3934f2f4-a9f4-44e2-b2e4-15e714f7ea2d","Handler":null,"Name":"","Endpoint":""} Jul 24 15:08:29.362546 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.362527 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 15:08:29.362693 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.362570 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 15:08:29.431140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.431101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:29.431325 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:29.431220 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:29.599137 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.599099 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" event={"ID":"cd8df5f7-9834-411d-a67a-99a1a3394206","Type":"ContainerStarted","Data":"ab72b915037caf61fb85fd3f22cc5754ee4cc7e560a8f6d4e03f2c23e4a68376"} Jul 24 15:08:29.600530 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.600505 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pcdg" event={"ID":"9b956246-0399-4cb5-9375-532d4f544fc6","Type":"ContainerStarted","Data":"43108b5a03068b9cdb183c12ae91bc2709bed21408d9675684b5a81757d27131"} Jul 24 15:08:29.619721 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:29.619612 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5pcdg" podStartSLOduration=6.03732617 podStartE2EDuration="23.619590219s" totalImagesPullingTime="17.582264049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:29.619071119 +0000 UTC m=+23.705723034" watchObservedRunningTime="2026-07-24 15:08:29.619590219 +0000 UTC m=+23.706242133" Jul 24 15:08:30.430476 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:30.430441 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:30.430476 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:30.430441 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:30.430802 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:30.430623 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:30.430802 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:30.430726 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:30.609912 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:30.609638 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:08:30.610334 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:30.610297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"8052d9dcf462abb2aa093cdd18cc7efd8c9bc79f83752b9650a132c8233a2e2d"} Jul 24 15:08:30.889232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:30.889194 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:30.889825 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:30.889807 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:31.430893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:31.430858 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:31.431124 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:31.430978 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:31.614878 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:31.614844 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" event={"ID":"cd8df5f7-9834-411d-a67a-99a1a3394206","Type":"ContainerStarted","Data":"545457cdf77f4f710fbd57d50ae0b19d50eee66a49b28372cab073f18b16f482"} Jul 24 15:08:31.637745 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:31.637686 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dttgs" podStartSLOduration=12.042770532 podStartE2EDuration="25.637663419s" totalImagesPullingTime="13.594892887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:31.637366867 +0000 UTC m=+25.724018780" watchObservedRunningTime="2026-07-24 15:08:31.637663419 +0000 UTC m=+25.724315333" Jul 24 15:08:32.430574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:32.430529 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:32.430767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:32.430585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:32.430767 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:32.430676 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:32.430883 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:32.430792 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:33.430925 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:33.430892 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:33.431547 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:33.430992 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:33.623679 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:33.623502 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:08:33.624026 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:33.623997 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"894703fe97bebffaf1deb91823886d5231f52bcd544d58839b571af36d39c192"} Jul 24 15:08:33.624297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:33.624277 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:33.624553 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:33.624538 2563 scope.go:117] "RemoveContainer" containerID="da55bbe29660ce8d4ae69315a0603e0a2529945ae24c80b5cea46f3b51fb4448" Jul 24 15:08:33.641135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:33.641110 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:34.430636 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.430598 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:34.430856 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.430644 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:34.430856 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:34.430721 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:34.430856 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:34.430829 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:34.631432 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.631361 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:08:34.631939 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.631826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" event={"ID":"b543d8cd-803e-4799-9c92-46c820f7dae2","Type":"ContainerStarted","Data":"30bea275a90cc5f899f06ccf649cb4820fd3a0894744dfacd5c0adbdb50296c4"} Jul 24 15:08:34.632677 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.632654 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:34.632804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.632724 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:34.634084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.634055 2563 generic.go:356] "Generic (PLEG): container finished" podID="84271708-62ea-4275-a46c-b8d088f96578" containerID="2db8c785671603f220f64443c8cc5bfde1205a7018decdcac70c4e7d701b3658" exitCode=0 Jul 24 15:08:34.634192 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.634098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerDied","Data":"2db8c785671603f220f64443c8cc5bfde1205a7018decdcac70c4e7d701b3658"} Jul 24 15:08:34.657474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.657421 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:08:34.700862 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:34.700814 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" podStartSLOduration=10.757882881 podStartE2EDuration="28.700798301s" totalImagesPullingTime="17.94291542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:34.674048997 +0000 UTC m=+28.760700910" watchObservedRunningTime="2026-07-24 15:08:34.700798301 +0000 UTC m=+28.787450215" Jul 24 15:08:35.184931 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.184894 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:35.185086 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.185073 2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 15:08:35.185562 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.185537 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bdctf" Jul 24 15:08:35.430848 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.430807 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:35.431008 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:35.430941 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:35.596525 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.596480 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5h4x9"] Jul 24 15:08:35.596692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.596618 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:35.596753 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:35.596729 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:35.599190 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.599152 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wb6w7"] Jul 24 15:08:35.603176 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.603150 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hsx6n"] Jul 24 15:08:35.603336 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.603305 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:35.603446 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:35.603423 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:35.636254 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:35.636214 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:35.636672 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:35.636358 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:36.640847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:36.640816 2563 generic.go:356] "Generic (PLEG): container finished" podID="84271708-62ea-4275-a46c-b8d088f96578" containerID="302f49d0d0bcbe14fd27e02f8525910735cf057ac685595c90a00d0b764a070b" exitCode=0 Jul 24 15:08:36.641254 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:36.640904 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerDied","Data":"302f49d0d0bcbe14fd27e02f8525910735cf057ac685595c90a00d0b764a070b"} Jul 24 15:08:37.430495 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:37.430315 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:37.430677 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:37.430324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:37.430677 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:37.430576 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:37.430677 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:37.430324 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:37.430677 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:37.430662 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:37.430821 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:37.430769 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:37.645947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:37.645906 2563 generic.go:356] "Generic (PLEG): container finished" podID="84271708-62ea-4275-a46c-b8d088f96578" containerID="64952d915fbbbad3c71eaf3814d483c51e6edb74c7df6b9133d09f4afb20da31" exitCode=0 Jul 24 15:08:37.646638 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:37.645954 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerDied","Data":"64952d915fbbbad3c71eaf3814d483c51e6edb74c7df6b9133d09f4afb20da31"} Jul 24 15:08:39.431019 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:39.430984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:39.431019 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:39.431010 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:39.431552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:39.431266 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:39.431552 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:39.431276 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb6w7" podUID="8c1b4722-2747-4d4a-8cb6-7deca0ce9676" Jul 24 15:08:39.431552 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:39.431384 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsx6n" podUID="3d99c649-459a-407b-aea5-83b0dd0b1120" Jul 24 15:08:39.431552 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:39.431468 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5h4x9" podUID="ff63c81a-f3de-41c9-8379-3719001ad778" Jul 24 15:08:40.064065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:40.064027 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:40.064289 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.064174 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:40.064289 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:40.064226 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:40.064289 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.064262 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:12.064224277 +0000 UTC m=+66.150876169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:40.064449 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.064327 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:40.064449 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.064379 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret podName:8c1b4722-2747-4d4a-8cb6-7deca0ce9676 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:12.064367727 +0000 UTC m=+66.151019618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret") pod "global-pull-secret-syncer-wb6w7" (UID: "8c1b4722-2747-4d4a-8cb6-7deca0ce9676") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:40.165302 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:40.165268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:40.165475 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.165405 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:40.165475 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.165421 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:40.165475 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.165432 2563 projected.go:196] Error preparing data for projected volume kube-api-access-cktl6 for pod openshift-network-diagnostics/network-check-target-5h4x9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:40.165604 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:40.165485 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6 podName:ff63c81a-f3de-41c9-8379-3719001ad778 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:12.165467769 +0000 UTC m=+66.252119678 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cktl6" (UniqueName: "kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6") pod "network-check-target-5h4x9" (UID: "ff63c81a-f3de-41c9-8379-3719001ad778") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:41.252408 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.252332 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-115.ec2.internal" event="NodeReady" Jul 24 15:08:41.252915 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.252491 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 15:08:41.290775 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.290741 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6579c999df-lmfnm"] Jul 24 15:08:41.293316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.293292 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.296201 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.296167 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 15:08:41.296366 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.296221 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 15:08:41.296366 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.296293 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 15:08:41.296632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.296616 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p5plk\"" Jul 24 15:08:41.301454 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.301416 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l9rg4"] Jul 24 15:08:41.303419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.303396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.304771 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.304747 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 15:08:41.307329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.307304 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 15:08:41.307443 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.307315 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 15:08:41.307493 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.307479 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkfwv\"" Jul 24 15:08:41.312479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.312454 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6579c999df-lmfnm"] Jul 24 15:08:41.321730 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.321696 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zw8rr"] Jul 24 15:08:41.323880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.323855 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.326767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.326745 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 15:08:41.326897 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.326743 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.327512 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.327490 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.327633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.327568 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-79w6c\"" Jul 24 15:08:41.334509 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.334461 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9rg4"] Jul 24 15:08:41.335343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.335318 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zw8rr"] Jul 24 15:08:41.374526 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374473 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-installation-pull-secrets\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.374717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-registry-certificates\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.374717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374585 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.374717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/326402ae-916b-475a-842b-f8e2543c3b02-ca-trust-extracted\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.374717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-image-registry-private-configuration\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.374717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-trusted-ca\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.375023 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374778 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdlc\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-kube-api-access-ksdlc\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.375023 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.374832 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-bound-sa-token\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.430706 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.430668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:08:41.430885 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.430668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:08:41.431035 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.430668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:08:41.433349 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.433318 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:08:41.433496 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.433423 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:08:41.433496 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.433453 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trkb8\"" Jul 24 15:08:41.433496 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.433317 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.433712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.433694 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d22wg\"" Jul 24 15:08:41.433828 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.433730 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.475181 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdlc\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-kube-api-access-ksdlc\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475368 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-config-volume\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.475368 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.475368 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-bound-sa-token\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475368 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5t28\" (UniqueName: \"kubernetes.io/projected/c3db899e-09ba-4b52-9b93-81c946f895e1-kube-api-access-l5t28\") pod \"ingress-canary-zw8rr\" (UID: \"c3db899e-09ba-4b52-9b93-81c946f895e1\") " pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.475368 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475346 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-tmp-dir\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-installation-pull-secrets\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrpqr\" (UniqueName: \"kubernetes.io/projected/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-kube-api-access-hrpqr\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-registry-certificates\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/326402ae-916b-475a-842b-f8e2543c3b02-ca-trust-extracted\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.475780 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-image-registry-private-configuration\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.475799 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-trusted-ca\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.475859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.475842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3db899e-09ba-4b52-9b93-81c946f895e1-cert\") pod \"ingress-canary-zw8rr\" (UID: \"c3db899e-09ba-4b52-9b93-81c946f895e1\") " pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.476233 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.475875 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:41.97584735 +0000 UTC m=+36.062499245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:08:41.476233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.476067 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/326402ae-916b-475a-842b-f8e2543c3b02-ca-trust-extracted\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.476364 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.476269 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-registry-certificates\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.476707 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.476687 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-trusted-ca\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.480807 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.480784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-installation-pull-secrets\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.480936 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.480783 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-image-registry-private-configuration\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.484906 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.484880 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-bound-sa-token\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.485030 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.484975 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdlc\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-kube-api-access-ksdlc\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.577061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrpqr\" (UniqueName: \"kubernetes.io/projected/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-kube-api-access-hrpqr\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.577269 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577110 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3db899e-09ba-4b52-9b93-81c946f895e1-cert\") pod \"ingress-canary-zw8rr\" (UID: \"c3db899e-09ba-4b52-9b93-81c946f895e1\") " pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.577269 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577142 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-config-volume\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.577269 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.577269 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5t28\" (UniqueName: \"kubernetes.io/projected/c3db899e-09ba-4b52-9b93-81c946f895e1-kube-api-access-l5t28\") pod \"ingress-canary-zw8rr\" (UID: \"c3db899e-09ba-4b52-9b93-81c946f895e1\") " pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.577269 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-tmp-dir\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.577534 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.577284 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:08:41.577534 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.577406 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:42.077383613 +0000 UTC m=+36.164035515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:08:41.577649 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577630 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-tmp-dir\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.577890 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.577867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-config-volume\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.579916 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.579895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3db899e-09ba-4b52-9b93-81c946f895e1-cert\") pod \"ingress-canary-zw8rr\" (UID: \"c3db899e-09ba-4b52-9b93-81c946f895e1\") " pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.589141 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.589107 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrpqr\" (UniqueName: \"kubernetes.io/projected/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-kube-api-access-hrpqr\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:41.592000 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.591967 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5t28\" (UniqueName: \"kubernetes.io/projected/c3db899e-09ba-4b52-9b93-81c946f895e1-kube-api-access-l5t28\") pod \"ingress-canary-zw8rr\" (UID: \"c3db899e-09ba-4b52-9b93-81c946f895e1\") " pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.635472 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.635428 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zw8rr" Jul 24 15:08:41.965748 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.965721 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zw8rr"] Jul 24 15:08:41.970291 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:08:41.970229 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3db899e_09ba_4b52_9b93_81c946f895e1.slice/crio-73b953e106bffe13cc360efe4ae659d8afbf85713b421ebe00e47f787870a021 WatchSource:0}: Error finding container 73b953e106bffe13cc360efe4ae659d8afbf85713b421ebe00e47f787870a021: Status 404 returned error can't find the container with id 73b953e106bffe13cc360efe4ae659d8afbf85713b421ebe00e47f787870a021 Jul 24 15:08:41.981744 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:41.981710 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:41.981887 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.981848 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:08:41.981887 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.981863 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:08:41.981994 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:41.981916 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:42.981900171 +0000 UTC m=+37.068552076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:08:42.082447 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:42.082384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:42.082529 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:42.082507 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:08:42.082574 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:42.082566 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:43.082552695 +0000 UTC m=+37.169204586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:08:42.661834 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:42.661799 2563 generic.go:356] "Generic (PLEG): container finished" podID="84271708-62ea-4275-a46c-b8d088f96578" containerID="6fd532845feb7dce1b48a53266bc7546ca99b46ec79ded11424732539bdde8f6" exitCode=0 Jul 24 15:08:42.662338 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:42.661882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerDied","Data":"6fd532845feb7dce1b48a53266bc7546ca99b46ec79ded11424732539bdde8f6"} Jul 24 15:08:42.662883 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:42.662863 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zw8rr" event={"ID":"c3db899e-09ba-4b52-9b93-81c946f895e1","Type":"ContainerStarted","Data":"73b953e106bffe13cc360efe4ae659d8afbf85713b421ebe00e47f787870a021"} Jul 24 15:08:42.989625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:42.989539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:42.989791 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:42.989711 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:08:42.989791 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:42.989733 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:08:42.989889 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:42.989804 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:44.989781768 +0000 UTC m=+39.076433674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:08:43.090227 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:43.090190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:43.090380 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:43.090346 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:08:43.090434 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:43.090411 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:45.090395588 +0000 UTC m=+39.177047479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:08:43.669096 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:43.668866 2563 generic.go:356] "Generic (PLEG): container finished" podID="84271708-62ea-4275-a46c-b8d088f96578" containerID="d3f4e89c58a501fa83ab3dd099f3708bc8e52f3e02022e91c30d2b81872762d9" exitCode=0 Jul 24 15:08:43.669096 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:43.668944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerDied","Data":"d3f4e89c58a501fa83ab3dd099f3708bc8e52f3e02022e91c30d2b81872762d9"} Jul 24 15:08:44.575168 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:44.575133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:44.675286 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:44.675238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:44.675868 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:44.675694 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" event={"ID":"84271708-62ea-4275-a46c-b8d088f96578","Type":"ContainerStarted","Data":"e19475ccf6361ad058b22218bf386b4377439ab0260ebeb6ee8f1cfa55b2e848"} Jul 24 15:08:44.700875 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:44.700813 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ccj4t" podStartSLOduration=10.192225533 podStartE2EDuration="38.700790797s" totalImagesPullingTime="28.508565264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:44.699729118 +0000 UTC m=+38.786381032" watchObservedRunningTime="2026-07-24 15:08:44.700790797 +0000 UTC m=+38.787442721" Jul 24 15:08:44.832609 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:44.832523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:44.899330 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:44.899286 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:45.007260 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:45.007204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:45.007423 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:45.007375 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:08:45.007423 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:45.007397 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:08:45.007498 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:45.007471 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:49.007453636 +0000 UTC m=+43.094105528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:08:45.108494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:45.108430 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:45.108598 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:45.108547 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:08:45.108637 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:45.108604 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:49.108591876 +0000 UTC m=+43.195243768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:08:45.679940 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:45.679900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zw8rr" event={"ID":"c3db899e-09ba-4b52-9b93-81c946f895e1","Type":"ContainerStarted","Data":"967cb4612d3f8058323201cc8da74bae248b19d958bc32552c05583bbaf2a985"} Jul 24 15:08:45.698960 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:45.698908 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zw8rr" podStartSLOduration=1.534986152 podStartE2EDuration="4.698894103s" totalImagesPullingTime="3.163907951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:45.698065132 +0000 UTC m=+39.784717045" watchObservedRunningTime="2026-07-24 15:08:45.698894103 +0000 UTC m=+39.785546015" Jul 24 15:08:49.037647 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:49.037608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:49.038026 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:49.037742 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:08:49.038026 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:49.037753 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:08:49.038026 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:49.037800 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:57.037787201 +0000 UTC m=+51.124439092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:08:49.138541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:49.138505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:49.138704 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:49.138650 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:08:49.138743 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:49.138714 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:57.138698909 +0000 UTC m=+51.225350801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:08:50.154569 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:50.154528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:50.258669 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:50.258626 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:50.359199 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:50.359163 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:51.036593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:51.036559 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:51.136698 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:51.136665 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:52.812085 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:52.812051 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:52.883821 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:52.883772 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:53.982351 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:53.982316 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:08:57.097808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:57.097769 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:08:57.098201 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:57.097902 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:08:57.098201 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:57.097935 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:08:57.098201 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:57.097991 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:13.09797824 +0000 UTC m=+67.184630136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:08:57.198459 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:08:57.198416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:08:57.198625 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:57.198551 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:08:57.198666 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:08:57.198629 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:13.198612127 +0000 UTC m=+67.285264019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:09:01.438518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:01.438480 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:01.536346 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:01.536309 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:01.636707 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:01.636671 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:01.705668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:01.705591 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:01.766788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:01.766755 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:02.448772 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:02.448742 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:02.549606 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:02.549572 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:02.652710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:02.652666 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:06.654410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:06.654381 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xg7m" Jul 24 15:09:06.940088 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:06.940001 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:07.033881 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:07.033842 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:07.147035 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:07.147003 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:07.254062 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:07.253965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:07.373957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:07.373922 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:08.812161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:08.812067 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:08.913087 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:08.913054 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:12.102897 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.102861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:09:12.103341 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.102961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:09:12.105811 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.105790 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:09:12.105864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.105833 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:09:12.114137 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:12.114107 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 15:09:12.114268 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:12.114190 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs podName:3d99c649-459a-407b-aea5-83b0dd0b1120 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:16.114172376 +0000 UTC m=+130.200824267 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs") pod "network-metrics-daemon-hsx6n" (UID: "3d99c649-459a-407b-aea5-83b0dd0b1120") : secret "metrics-daemon-secret" not found Jul 24 15:09:12.117864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.117841 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c1b4722-2747-4d4a-8cb6-7deca0ce9676-original-pull-secret\") pod \"global-pull-secret-syncer-wb6w7\" (UID: \"8c1b4722-2747-4d4a-8cb6-7deca0ce9676\") " pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:09:12.203521 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.203484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:09:12.207316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.207293 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:09:12.216729 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.216708 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:09:12.226738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.226707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktl6\" (UniqueName: \"kubernetes.io/projected/ff63c81a-f3de-41c9-8379-3719001ad778-kube-api-access-cktl6\") pod \"network-check-target-5h4x9\" (UID: \"ff63c81a-f3de-41c9-8379-3719001ad778\") " pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:09:12.349980 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.349943 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb6w7" Jul 24 15:09:12.358714 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.358688 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d22wg\"" Jul 24 15:09:12.366715 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.366687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:09:12.508222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.508194 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wb6w7"] Jul 24 15:09:12.511988 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:09:12.511959 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1b4722_2747_4d4a_8cb6_7deca0ce9676.slice/crio-cb120f121194493674eccba6112044b21e6cebf25936bf72626c1d91f96652c7 WatchSource:0}: Error finding container cb120f121194493674eccba6112044b21e6cebf25936bf72626c1d91f96652c7: Status 404 returned error can't find the container with id cb120f121194493674eccba6112044b21e6cebf25936bf72626c1d91f96652c7 Jul 24 15:09:12.541235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.541204 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5h4x9"] Jul 24 15:09:12.544349 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:09:12.544320 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff63c81a_f3de_41c9_8379_3719001ad778.slice/crio-2943d74b03ce742982e5bcaa7264848f0dc1eef6eb5f804293d3d9319413d7c6 WatchSource:0}: Error finding container 2943d74b03ce742982e5bcaa7264848f0dc1eef6eb5f804293d3d9319413d7c6: Status 404 returned error can't find the container with id 2943d74b03ce742982e5bcaa7264848f0dc1eef6eb5f804293d3d9319413d7c6 Jul 24 15:09:12.738287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.738179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5h4x9" event={"ID":"ff63c81a-f3de-41c9-8379-3719001ad778","Type":"ContainerStarted","Data":"2943d74b03ce742982e5bcaa7264848f0dc1eef6eb5f804293d3d9319413d7c6"} Jul 24 15:09:12.739279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:12.739238 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wb6w7" event={"ID":"8c1b4722-2747-4d4a-8cb6-7deca0ce9676","Type":"ContainerStarted","Data":"cb120f121194493674eccba6112044b21e6cebf25936bf72626c1d91f96652c7"} Jul 24 15:09:13.110179 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:13.110141 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:09:13.110612 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:13.110297 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:09:13.110612 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:13.110317 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:09:13.110612 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:13.110372 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:45.11035558 +0000 UTC m=+99.197007491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:09:13.210679 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:13.210642 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:09:13.210931 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:13.210794 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:13.210931 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:13.210847 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:45.210834793 +0000 UTC m=+99.297486685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:09:13.750102 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:13.750060 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:13.830908 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:13.830862 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:15.749458 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:15.749417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5h4x9" event={"ID":"ff63c81a-f3de-41c9-8379-3719001ad778","Type":"ContainerStarted","Data":"1c170e7f44bf97a6e2ae4209bf802baf188a4d54e0af56651dd06e7d01a20b72"} Jul 24 15:09:15.749895 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:15.749575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:09:15.767406 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:15.767298 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5h4x9" podStartSLOduration=66.801735627 podStartE2EDuration="1m9.767235406s" totalImagesPullingTime="2.965499779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:15.76676322 +0000 UTC m=+69.853415138" watchObservedRunningTime="2026-07-24 15:09:15.767235406 +0000 UTC m=+69.853887319" Jul 24 15:09:17.756752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:17.756717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wb6w7" event={"ID":"8c1b4722-2747-4d4a-8cb6-7deca0ce9676","Type":"ContainerStarted","Data":"4127ad75c8aa0a38a7f205233c9718c7875f4cb493a8a558b10d81fc293f4dc2"} Jul 24 15:09:17.779524 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:17.779451 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wb6w7" podStartSLOduration=67.220496151 podStartE2EDuration="1m11.779430414s" totalImagesPullingTime="4.558934263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:17.779380492 +0000 UTC m=+71.866032406" watchObservedRunningTime="2026-07-24 15:09:17.779430414 +0000 UTC m=+71.866082329" Jul 24 15:09:18.097870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:18.097815 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:19.365347 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:19.365306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:28.229455 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:28.229414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:28.320983 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:28.320945 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:28.389886 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:28.389855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:40.547944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:40.547902 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:40.609793 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:40.609738 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:40.664355 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:40.664323 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:45.142440 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:45.142375 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:09:45.142771 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:45.142524 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:09:45.142771 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:45.142545 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6579c999df-lmfnm: secret "image-registry-tls" not found Jul 24 15:09:45.142771 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:45.142601 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls podName:326402ae-916b-475a-842b-f8e2543c3b02 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:49.142586581 +0000 UTC m=+163.229238478 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls") pod "image-registry-6579c999df-lmfnm" (UID: "326402ae-916b-475a-842b-f8e2543c3b02") : secret "image-registry-tls" not found Jul 24 15:09:45.243539 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:45.243500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:09:45.243646 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:45.243632 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:45.243692 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:45.243684 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls podName:f157fb48-58c1-4f43-bc15-b8dd74dfcc02 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:49.243672483 +0000 UTC m=+163.330324374 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls") pod "dns-default-l9rg4" (UID: "f157fb48-58c1-4f43-bc15-b8dd74dfcc02") : secret "dns-default-metrics-tls" not found Jul 24 15:09:46.755334 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:46.755302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5h4x9" Jul 24 15:09:52.858329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.858286 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f"] Jul 24 15:09:52.861462 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.861435 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:52.864362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.864338 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 15:09:52.864663 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.864645 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7hd69\"" Jul 24 15:09:52.864774 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.864649 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:09:52.865287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.865269 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 15:09:52.865545 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.865528 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 15:09:52.870799 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.870736 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz"] Jul 24 15:09:52.872642 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.872621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:52.875440 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.875417 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 15:09:52.875872 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.875599 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 15:09:52.875872 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.875740 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 15:09:52.876060 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.875879 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sffm6\"" Jul 24 15:09:52.876060 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.875938 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:09:52.876817 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.876802 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f"] Jul 24 15:09:52.907104 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.907066 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz"] Jul 24 15:09:52.965890 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.965859 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh"] Jul 24 15:09:52.967728 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.967706 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:52.970417 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.970396 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 15:09:52.970717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.970699 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 15:09:52.970782 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.970765 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8xsp9\"" Jul 24 15:09:52.970818 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.970793 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:09:52.998518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.998481 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:52.998518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.998529 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8btg\" (UniqueName: \"kubernetes.io/projected/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-kube-api-access-b8btg\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:52.998746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.998563 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:52.998746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.998647 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jphkc\" (UniqueName: \"kubernetes.io/projected/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-kube-api-access-jphkc\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:52.998746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.998684 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:52.998746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:52.998706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-config\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.015109 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.015075 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh"] Jul 24 15:09:53.057402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.057364 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt"] Jul 24 15:09:53.059084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.059067 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" Jul 24 15:09:53.062063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.062037 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-v87bl\"" Jul 24 15:09:53.099948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.099914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.099966 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jphkc\" (UniqueName: \"kubernetes.io/projected/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-kube-api-access-jphkc\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.099985 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-config\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100074 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27625\" (UniqueName: \"kubernetes.io/projected/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-kube-api-access-27625\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:53.100140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8btg\" (UniqueName: \"kubernetes.io/projected/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-kube-api-access-b8btg\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.100659 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.100743 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.100689 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-config\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.102237 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.102208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-serving-cert\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.102356 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.102267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.109066 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.109007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jphkc\" (UniqueName: \"kubernetes.io/projected/8530a5e5-08c1-4d38-9d9c-429f9bf677c5-kube-api-access-jphkc\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f\" (UID: \"8530a5e5-08c1-4d38-9d9c-429f9bf677c5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.109173 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.109156 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8btg\" (UniqueName: \"kubernetes.io/projected/8d65cb26-ec51-4d21-9839-2bc01a1f7d47-kube-api-access-b8btg\") pod \"service-ca-operator-5d6c75f84f-jw8gz\" (UID: \"8d65cb26-ec51-4d21-9839-2bc01a1f7d47\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.126340 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.126310 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt"] Jul 24 15:09:53.170697 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.170654 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" Jul 24 15:09:53.182320 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.182291 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" Jul 24 15:09:53.200648 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.200614 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:53.200823 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.200670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27625\" (UniqueName: \"kubernetes.io/projected/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-kube-api-access-27625\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:53.200823 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.200731 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2q6j\" (UniqueName: \"kubernetes.io/projected/57ed3663-fe86-448a-8a5a-a2c23296fffe-kube-api-access-v2q6j\") pod \"network-check-source-7d88cb55d8-9jjrt\" (UID: \"57ed3663-fe86-448a-8a5a-a2c23296fffe\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" Jul 24 15:09:53.200823 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:53.200777 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:09:53.200964 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:53.200862 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls podName:2ea4a8b4-b74b-4140-a377-1e75ff7354c9 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:53.70084082 +0000 UTC m=+107.787492724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls") pod "cluster-samples-operator-6748489656-sdjrh" (UID: "2ea4a8b4-b74b-4140-a377-1e75ff7354c9") : secret "samples-operator-tls" not found Jul 24 15:09:53.210330 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.210299 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27625\" (UniqueName: \"kubernetes.io/projected/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-kube-api-access-27625\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:53.225612 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.225576 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:53.301941 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.301910 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2q6j\" (UniqueName: \"kubernetes.io/projected/57ed3663-fe86-448a-8a5a-a2c23296fffe-kube-api-access-v2q6j\") pod \"network-check-source-7d88cb55d8-9jjrt\" (UID: \"57ed3663-fe86-448a-8a5a-a2c23296fffe\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" Jul 24 15:09:53.303614 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.303589 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f"] Jul 24 15:09:53.307177 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:09:53.307145 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8530a5e5_08c1_4d38_9d9c_429f9bf677c5.slice/crio-abb1f647f1444cbd79277d5f2251eb13e0807a094afdf6a8b08efe735b98da18 WatchSource:0}: Error finding container abb1f647f1444cbd79277d5f2251eb13e0807a094afdf6a8b08efe735b98da18: Status 404 returned error can't find the container with id abb1f647f1444cbd79277d5f2251eb13e0807a094afdf6a8b08efe735b98da18 Jul 24 15:09:53.311123 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.311093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:53.314648 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.314628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2q6j\" (UniqueName: \"kubernetes.io/projected/57ed3663-fe86-448a-8a5a-a2c23296fffe-kube-api-access-v2q6j\") pod \"network-check-source-7d88cb55d8-9jjrt\" (UID: \"57ed3663-fe86-448a-8a5a-a2c23296fffe\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" Jul 24 15:09:53.322645 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.322607 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz"] Jul 24 15:09:53.327211 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:09:53.327185 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d65cb26_ec51_4d21_9839_2bc01a1f7d47.slice/crio-e5e49b04e77f6e23f05f8c7b81bf194a65e14f9873dc41bb988f1c4500fe1e12 WatchSource:0}: Error finding container e5e49b04e77f6e23f05f8c7b81bf194a65e14f9873dc41bb988f1c4500fe1e12: Status 404 returned error can't find the container with id e5e49b04e77f6e23f05f8c7b81bf194a65e14f9873dc41bb988f1c4500fe1e12 Jul 24 15:09:53.367517 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.367425 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" Jul 24 15:09:53.488757 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.488724 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt"] Jul 24 15:09:53.491579 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:09:53.491548 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ed3663_fe86_448a_8a5a_a2c23296fffe.slice/crio-f8647b2a66f9a77cb04766d4198bf4952d26ba129b3a667962a2da0b86af4be9 WatchSource:0}: Error finding container f8647b2a66f9a77cb04766d4198bf4952d26ba129b3a667962a2da0b86af4be9: Status 404 returned error can't find the container with id f8647b2a66f9a77cb04766d4198bf4952d26ba129b3a667962a2da0b86af4be9 Jul 24 15:09:53.707185 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.707078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:53.707396 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:53.707291 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:09:53.707396 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:53.707369 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls podName:2ea4a8b4-b74b-4140-a377-1e75ff7354c9 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:54.707349565 +0000 UTC m=+108.794001457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls") pod "cluster-samples-operator-6748489656-sdjrh" (UID: "2ea4a8b4-b74b-4140-a377-1e75ff7354c9") : secret "samples-operator-tls" not found Jul 24 15:09:53.843444 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.843398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" event={"ID":"57ed3663-fe86-448a-8a5a-a2c23296fffe","Type":"ContainerStarted","Data":"dfa507111542541fa800dbf11da131f4a50f42971ce1eea71b4dad5b71c10f4b"} Jul 24 15:09:53.843742 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.843500 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" event={"ID":"57ed3663-fe86-448a-8a5a-a2c23296fffe","Type":"ContainerStarted","Data":"f8647b2a66f9a77cb04766d4198bf4952d26ba129b3a667962a2da0b86af4be9"} Jul 24 15:09:53.844556 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.844529 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" event={"ID":"8530a5e5-08c1-4d38-9d9c-429f9bf677c5","Type":"ContainerStarted","Data":"abb1f647f1444cbd79277d5f2251eb13e0807a094afdf6a8b08efe735b98da18"} Jul 24 15:09:53.845552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.845534 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" event={"ID":"8d65cb26-ec51-4d21-9839-2bc01a1f7d47","Type":"ContainerStarted","Data":"e5e49b04e77f6e23f05f8c7b81bf194a65e14f9873dc41bb988f1c4500fe1e12"} Jul 24 15:09:53.861632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:53.861580 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-9jjrt" podStartSLOduration=0.861559944 podStartE2EDuration="861.559944ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:53.860724613 +0000 UTC m=+107.947376526" watchObservedRunningTime="2026-07-24 15:09:53.861559944 +0000 UTC m=+107.948211835" Jul 24 15:09:54.715833 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:54.715794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:54.716004 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:54.715946 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:09:54.716045 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:54.716016 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls podName:2ea4a8b4-b74b-4140-a377-1e75ff7354c9 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:56.715999145 +0000 UTC m=+110.802651036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls") pod "cluster-samples-operator-6748489656-sdjrh" (UID: "2ea4a8b4-b74b-4140-a377-1e75ff7354c9") : secret "samples-operator-tls" not found Jul 24 15:09:55.251888 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:55.251846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:55.351270 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:55.351216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:56.628735 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.628697 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jcm7r"] Jul 24 15:09:56.630659 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.630634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.633105 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.633067 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 15:09:56.633105 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.633094 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvtjc\"" Jul 24 15:09:56.634147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.634114 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 15:09:56.634147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.634133 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 15:09:56.634410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.634394 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 15:09:56.651103 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.651069 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jcm7r"] Jul 24 15:09:56.678389 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.678347 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:56.734461 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.734422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn6c\" (UniqueName: \"kubernetes.io/projected/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-api-access-gvn6c\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.734461 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.734465 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b5ca429-11b2-4d36-9222-8f3f71530f7d-data-volume\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.734667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.734491 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.734667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.734602 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:09:56.734667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.734625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.734667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.734642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b5ca429-11b2-4d36-9222-8f3f71530f7d-crio-socket\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.734828 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:56.734804 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:09:56.734898 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:56.734888 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls podName:2ea4a8b4-b74b-4140-a377-1e75ff7354c9 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.734866589 +0000 UTC m=+114.821518485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls") pod "cluster-samples-operator-6748489656-sdjrh" (UID: "2ea4a8b4-b74b-4140-a377-1e75ff7354c9") : secret "samples-operator-tls" not found Jul 24 15:09:56.779905 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.779861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:56.835836 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.835795 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b5ca429-11b2-4d36-9222-8f3f71530f7d-data-volume\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.835836 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.835836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.836074 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:56.835956 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:09:56.836074 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:56.836016 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:09:57.335996184 +0000 UTC m=+111.422648076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : secret "insights-runtime-extractor-tls" not found Jul 24 15:09:56.836074 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.836059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.836235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.836076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b5ca429-11b2-4d36-9222-8f3f71530f7d-crio-socket\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.836235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.836111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn6c\" (UniqueName: \"kubernetes.io/projected/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-api-access-gvn6c\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.836235 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:56.836125 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:09:56.836235 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:56.836193 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:09:57.3361753 +0000 UTC m=+111.422827192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : configmap "kube-rbac-proxy" not found Jul 24 15:09:56.836235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.836195 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b5ca429-11b2-4d36-9222-8f3f71530f7d-crio-socket\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.836235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.836206 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b5ca429-11b2-4d36-9222-8f3f71530f7d-data-volume\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.846209 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.846184 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn6c\" (UniqueName: \"kubernetes.io/projected/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-api-access-gvn6c\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:56.858318 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.858282 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" event={"ID":"8530a5e5-08c1-4d38-9d9c-429f9bf677c5","Type":"ContainerStarted","Data":"a18c1b96cef709920753e2e5ccce85fb81b8dcf2055a9de420c8147b98a7f086"} Jul 24 15:09:56.859859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.859823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" event={"ID":"8d65cb26-ec51-4d21-9839-2bc01a1f7d47","Type":"ContainerStarted","Data":"e4d804fa68d6e9578f2c5bd03ed24d46ee9cb32a4d1e4174802e55d425969e58"} Jul 24 15:09:56.887254 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.887140 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" podStartSLOduration=2.242361198 podStartE2EDuration="4.887124927s" totalImagesPullingTime="2.644763729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:56.886583058 +0000 UTC m=+110.973234971" watchObservedRunningTime="2026-07-24 15:09:56.887124927 +0000 UTC m=+110.973776882" Jul 24 15:09:56.909422 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:56.909358 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" podStartSLOduration=2.283221552 podStartE2EDuration="4.909334295s" totalImagesPullingTime="2.626112743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:56.908061756 +0000 UTC m=+110.994713671" watchObservedRunningTime="2026-07-24 15:09:56.909334295 +0000 UTC m=+110.995986208" Jul 24 15:09:57.173309 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.173216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:57.237603 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.237571 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:57.341732 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.341692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:57.341944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.341789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:57.341944 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:57.341864 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:09:57.341944 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:57.341868 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:09:57.341944 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:57.341917 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:09:58.341903704 +0000 UTC m=+112.428555596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : configmap "kube-rbac-proxy" not found Jul 24 15:09:57.341944 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:57.341930 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:09:58.341924068 +0000 UTC m=+112.428575960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : secret "insights-runtime-extractor-tls" not found Jul 24 15:09:57.401031 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.400996 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:57.514999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.514965 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:57.604264 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.604205 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:57.668998 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.668959 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:57.900088 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:57.899997 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:58.007451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:58.007405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:58.350380 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:58.350333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:58.350555 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:58.350409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:09:58.350555 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:58.350466 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:09:58.350555 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:58.350538 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.350522275 +0000 UTC m=+114.437174167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : configmap "kube-rbac-proxy" not found Jul 24 15:09:58.350555 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:58.350540 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:09:58.350696 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:09:58.350596 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.350581453 +0000 UTC m=+114.437233347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : secret "insights-runtime-extractor-tls" not found Jul 24 15:09:58.375442 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:58.375419 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-656vw_40253b47-eaff-437d-a540-2526e5e4f40d/dns-node-resolver/0.log" Jul 24 15:09:59.588701 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:59.588671 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9bzd_d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0/node-ca/0.log" Jul 24 15:09:59.648578 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:59.648541 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:09:59.765893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:09:59.765855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:00.344656 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.344620 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:00.369642 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.369602 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:00.369810 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.369657 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:00.369810 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:00.369751 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:10:00.369810 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:00.369781 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:10:00.369976 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:00.369826 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:04.369806275 +0000 UTC m=+118.456458167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : configmap "kube-rbac-proxy" not found Jul 24 15:10:00.369976 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:00.369844 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:04.369835779 +0000 UTC m=+118.456487670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : secret "insights-runtime-extractor-tls" not found Jul 24 15:10:00.436203 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.436166 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:00.577957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.577918 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zw8rr_c3db899e-09ba-4b52-9b93-81c946f895e1/serve-healthcheck-canary/0.log" Jul 24 15:10:00.695432 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.695340 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:00.773715 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:00.773681 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:10:00.773843 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:00.773797 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:10:00.773896 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:00.773857 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls podName:2ea4a8b4-b74b-4140-a377-1e75ff7354c9 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:08.773840902 +0000 UTC m=+122.860492793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls") pod "cluster-samples-operator-6748489656-sdjrh" (UID: "2ea4a8b4-b74b-4140-a377-1e75ff7354c9") : secret "samples-operator-tls" not found Jul 24 15:10:03.829788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:03.829743 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:04.219552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.219450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:04.327433 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.327396 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:04.405134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.405094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:04.405348 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.405152 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:04.405348 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:04.405221 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:10:04.405348 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:04.405268 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:10:04.405348 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:04.405315 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:12.405295338 +0000 UTC m=+126.491947237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : secret "insights-runtime-extractor-tls" not found Jul 24 15:10:04.405348 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:04.405338 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:12.405329209 +0000 UTC m=+126.491981102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : configmap "kube-rbac-proxy" not found Jul 24 15:10:04.480185 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.480103 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:04.595309 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.595278 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:04.703606 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.703571 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:04.803321 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:04.803289 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:05.082131 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:05.082044 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:05.492822 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:05.492723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:06.075071 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:06.075042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:06.679113 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:06.679076 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:06.895919 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:06.895882 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:07.121973 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:07.121940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:07.280171 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:07.280134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:07.478133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:07.478046 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:07.799054 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:07.799016 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:08.289063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.289028 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:08.686143 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.686049 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:08.838782 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.838747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:10:08.841183 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.841158 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ea4a8b4-b74b-4140-a377-1e75ff7354c9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sdjrh\" (UID: \"2ea4a8b4-b74b-4140-a377-1e75ff7354c9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:10:08.878996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.878964 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8xsp9\"" Jul 24 15:10:08.887021 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.886999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" Jul 24 15:10:08.922291 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:08.922230 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:09.004147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:09.004114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:09.015584 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:09.015557 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh"] Jul 24 15:10:09.077892 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:09.077863 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:09.284974 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:09.284940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:09.837333 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:09.837296 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:09.897821 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:09.897786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" event={"ID":"2ea4a8b4-b74b-4140-a377-1e75ff7354c9","Type":"ContainerStarted","Data":"6745fa001aba5d60f2a98c2669f7253915b235cc71aa3093b03eda8d5a5c430d"} Jul 24 15:10:10.272173 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:10.272134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:10.669865 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:10.669773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:10.878172 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:10.878134 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:11.072767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.072736 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:11.175535 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.175492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:11.280287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.280238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:11.876702 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.876667 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:11.905667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.905632 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" event={"ID":"2ea4a8b4-b74b-4140-a377-1e75ff7354c9","Type":"ContainerStarted","Data":"158cb801100f89694e0386a26d0c989b68fcf907bd4b5a88a6dc8ab353bbc821"} Jul 24 15:10:11.905802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.905696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" event={"ID":"2ea4a8b4-b74b-4140-a377-1e75ff7354c9","Type":"ContainerStarted","Data":"54e01adf7ecfe9711c6bae60005e9c293d8366e5443624eacd3303ba7289c8a8"} Jul 24 15:10:11.925122 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:11.925075 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sdjrh" podStartSLOduration=17.937295392 podStartE2EDuration="19.925059476s" totalImagesPullingTime="1.987764084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:11.923934122 +0000 UTC m=+126.010586035" watchObservedRunningTime="2026-07-24 15:10:11.925059476 +0000 UTC m=+126.011711447" Jul 24 15:10:12.396823 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:12.396785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:12.466299 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:12.466255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:12.466446 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:12.466323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:12.466502 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:12.466473 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:10:12.466565 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:12.466551 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls podName:0b5ca429-11b2-4d36-9222-8f3f71530f7d nodeName:}" failed. No retries permitted until 2026-07-24 15:10:28.466528466 +0000 UTC m=+142.553180367 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcm7r" (UID: "0b5ca429-11b2-4d36-9222-8f3f71530f7d") : secret "insights-runtime-extractor-tls" not found Jul 24 15:10:12.466772 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:12.466752 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b5ca429-11b2-4d36-9222-8f3f71530f7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:12.497638 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:12.497580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:12.880853 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:12.880808 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:13.306531 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:13.306502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:13.486874 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:13.486840 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:14.078130 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:14.078096 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:14.283825 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:14.283785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:14.476218 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:14.476115 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:15.009494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:15.009453 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:15.109535 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:15.109497 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:15.216017 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:15.215987 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:15.349914 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:15.349823 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:15.693808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:15.693724 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:15.792084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:15.792053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:16.083111 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.083074 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:16.157048 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.157020 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:16.190129 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.190092 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:10:16.192409 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.192389 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d99c649-459a-407b-aea5-83b0dd0b1120-metrics-certs\") pod \"network-metrics-daemon-hsx6n\" (UID: \"3d99c649-459a-407b-aea5-83b0dd0b1120\") " pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:10:16.245047 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.245008 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trkb8\"" Jul 24 15:10:16.253175 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.253143 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsx6n" Jul 24 15:10:16.379224 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.379143 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hsx6n"] Jul 24 15:10:16.382517 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:16.382491 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d99c649_459a_407b_aea5_83b0dd0b1120.slice/crio-55193d74130d088d03c5a8c19297632b98cab4152e6b14274a53643af0608252 WatchSource:0}: Error finding container 55193d74130d088d03c5a8c19297632b98cab4152e6b14274a53643af0608252: Status 404 returned error can't find the container with id 55193d74130d088d03c5a8c19297632b98cab4152e6b14274a53643af0608252 Jul 24 15:10:16.686998 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.686900 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:16.895133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.895090 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:16.922189 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.922149 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsx6n" event={"ID":"3d99c649-459a-407b-aea5-83b0dd0b1120","Type":"ContainerStarted","Data":"55193d74130d088d03c5a8c19297632b98cab4152e6b14274a53643af0608252"} Jul 24 15:10:16.971718 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:16.971613 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:17.101989 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:17.101949 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:17.926893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:17.926855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsx6n" event={"ID":"3d99c649-459a-407b-aea5-83b0dd0b1120","Type":"ContainerStarted","Data":"6b1240a630ed9d6ba8b151ff521ac95e6364b507eb74d0d808768efa85cfa572"} Jul 24 15:10:17.927106 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:17.926916 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsx6n" event={"ID":"3d99c649-459a-407b-aea5-83b0dd0b1120","Type":"ContainerStarted","Data":"e3c9e34dd4f50d6e3434ba563f275a99b948d200e22b4feb3c2d86155688a5ea"} Jul 24 15:10:17.949439 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:17.949383 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hsx6n" podStartSLOduration=131.118612752 podStartE2EDuration="2m11.949364119s" totalImagesPullingTime="830.751367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:17.947803739 +0000 UTC m=+132.034455652" watchObservedRunningTime="2026-07-24 15:10:17.949364119 +0000 UTC m=+132.036016031" Jul 24 15:10:22.822285 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:22.822236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:22.884411 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:22.884373 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:24.915940 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:24.915905 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:25.030715 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.030673 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:25.878926 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.878892 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl"] Jul 24 15:10:25.883101 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.883074 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.891738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.891706 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 15:10:25.891902 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.891855 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 15:10:25.891902 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.891874 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 15:10:25.892151 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.892135 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 15:10:25.892635 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.892618 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 15:10:25.892635 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.892629 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 15:10:25.910169 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.910136 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 15:10:25.961031 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.960990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2946d44c-0900-4376-a145-236969a5d9cb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.961031 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.961028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.961479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.961051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-hub\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.961479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.961072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-ca\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.961479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.961158 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.961479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.961191 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqp6z\" (UniqueName: \"kubernetes.io/projected/2946d44c-0900-4376-a145-236969a5d9cb-kube-api-access-jqp6z\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:25.967716 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:25.967684 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl"] Jul 24 15:10:26.061767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.061732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2946d44c-0900-4376-a145-236969a5d9cb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.061767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.061770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.062028 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.061799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-hub\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.062028 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.061836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-ca\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.062133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.062021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.062133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.062054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqp6z\" (UniqueName: \"kubernetes.io/projected/2946d44c-0900-4376-a145-236969a5d9cb-kube-api-access-jqp6z\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.062693 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.062659 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2946d44c-0900-4376-a145-236969a5d9cb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.063013 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.062987 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:26.064636 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.064608 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-ca\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.064765 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.064748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.064852 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.064808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.065015 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.064997 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2946d44c-0900-4376-a145-236969a5d9cb-hub\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.072356 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.072324 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqp6z\" (UniqueName: \"kubernetes.io/projected/2946d44c-0900-4376-a145-236969a5d9cb-kube-api-access-jqp6z\") pod \"cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl\" (UID: \"2946d44c-0900-4376-a145-236969a5d9cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.197710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.197625 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:26.201325 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.201299 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" Jul 24 15:10:26.341234 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.341192 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl"] Jul 24 15:10:26.344697 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:26.344672 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2946d44c_0900_4376_a145_236969a5d9cb.slice/crio-2956d05800f93d70b2946c90e7532a549ebe826a584c67d221ca26eb8b4bce4e WatchSource:0}: Error finding container 2956d05800f93d70b2946c90e7532a549ebe826a584c67d221ca26eb8b4bce4e: Status 404 returned error can't find the container with id 2956d05800f93d70b2946c90e7532a549ebe826a584c67d221ca26eb8b4bce4e Jul 24 15:10:26.867682 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.867641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:26.944534 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.944496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:26.958814 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:26.958779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" event={"ID":"2946d44c-0900-4376-a145-236969a5d9cb","Type":"ContainerStarted","Data":"2956d05800f93d70b2946c90e7532a549ebe826a584c67d221ca26eb8b4bce4e"} Jul 24 15:10:27.883989 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:27.883953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:27.980813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:27.980775 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:28.061014 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:28.060960 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:28.398519 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:28.398457 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:28.484887 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:28.484851 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:28.487917 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:28.487856 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b5ca429-11b2-4d36-9222-8f3f71530f7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcm7r\" (UID: \"0b5ca429-11b2-4d36-9222-8f3f71530f7d\") " pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:28.744372 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:28.744285 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvtjc\"" Jul 24 15:10:28.751767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:28.751737 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jcm7r" Jul 24 15:10:29.156962 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:29.156914 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jcm7r"] Jul 24 15:10:30.197294 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:30.197253 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5ca429_11b2_4d36_9222_8f3f71530f7d.slice/crio-0182007c0314e438fcf34e9c059e7eacafcb5f263a4d920acf3c7c4fcad88e78 WatchSource:0}: Error finding container 0182007c0314e438fcf34e9c059e7eacafcb5f263a4d920acf3c7c4fcad88e78: Status 404 returned error can't find the container with id 0182007c0314e438fcf34e9c059e7eacafcb5f263a4d920acf3c7c4fcad88e78 Jul 24 15:10:30.977747 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:30.977644 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" event={"ID":"2946d44c-0900-4376-a145-236969a5d9cb","Type":"ContainerStarted","Data":"a4a396f7c00e583739fa598f0290b9f6113705651cf4e9ec86e5ae4ce93c7497"} Jul 24 15:10:30.977747 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:30.977698 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" event={"ID":"2946d44c-0900-4376-a145-236969a5d9cb","Type":"ContainerStarted","Data":"3b92d8d72444181c632c9ff3177b442b5872a23a6dc95d4ce280f308a8693a60"} Jul 24 15:10:30.977747 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:30.977717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" event={"ID":"2946d44c-0900-4376-a145-236969a5d9cb","Type":"ContainerStarted","Data":"2a2f4e8d7d45aea27cdaad7f068530708b15d050aca8236edddbeb33a511ffd2"} Jul 24 15:10:30.979695 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:30.979648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcm7r" event={"ID":"0b5ca429-11b2-4d36-9222-8f3f71530f7d","Type":"ContainerStarted","Data":"0182007c0314e438fcf34e9c059e7eacafcb5f263a4d920acf3c7c4fcad88e78"} Jul 24 15:10:31.000579 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:31.000501 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9cb7c47f-k2mkl" podStartSLOduration=1.973221298 podStartE2EDuration="6.000475075s" totalImagesPullingTime="4.027253777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:30.998328114 +0000 UTC m=+145.084980026" watchObservedRunningTime="2026-07-24 15:10:31.000475075 +0000 UTC m=+145.087126976" Jul 24 15:10:32.079923 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:32.079887 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:32.155812 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:32.155776 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:32.541211 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:32.541158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:33.527060 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:33.527016 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:33.604372 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:33.604342 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:33.929808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:33.929665 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:34.029997 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:34.029964 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:34.476872 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:34.476829 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:34.584220 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:34.584183 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:35.796798 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:35.796758 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:35.870153 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:35.869994 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:35.999018 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:35.998981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcm7r" event={"ID":"0b5ca429-11b2-4d36-9222-8f3f71530f7d","Type":"ContainerStarted","Data":"ff3c9534b66472eea3907ad49ddb3ebcff24b8ae21220e081c7479302746c44c"} Jul 24 15:10:37.527894 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:37.527844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:38.634234 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:38.634139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:38.744325 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:38.744287 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:38.932943 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:38.932850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:39.010935 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:39.010900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcm7r" event={"ID":"0b5ca429-11b2-4d36-9222-8f3f71530f7d","Type":"ContainerStarted","Data":"cba025cd7605dc0f77c9869e1c1957259f0655c7d65f40907c780ecd4a7ccb07"} Jul 24 15:10:39.033123 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:39.033082 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:39.339537 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:39.339497 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:39.800763 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:39.800720 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:39.993236 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:39.993160 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tsvt7"] Jul 24 15:10:39.996767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:39.996743 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.002335 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002307 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 15:10:40.002335 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002329 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 15:10:40.002543 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002314 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 15:10:40.002807 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:40.002930 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002846 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hs9mz\"" Jul 24 15:10:40.002930 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002858 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 15:10:40.003048 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.002943 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 15:10:40.003102 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.003047 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 15:10:40.089461 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-sys\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089461 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0c31-de73-4d43-8635-82e60062a477-metrics-client-ca\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089690 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089511 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-tls\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089690 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089536 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-wtmp\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089690 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-root\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089690 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089657 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nh7r\" (UniqueName: \"kubernetes.io/projected/12fa0c31-de73-4d43-8635-82e60062a477-kube-api-access-2nh7r\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-textfile\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089729 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.089847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.089756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.170286 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.170178 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:40.190289 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-tls\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190298 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-wtmp\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-root\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nh7r\" (UniqueName: \"kubernetes.io/projected/12fa0c31-de73-4d43-8635-82e60062a477-kube-api-access-2nh7r\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190420 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-textfile\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190443 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-wtmp\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190551 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-sys\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0c31-de73-4d43-8635-82e60062a477-metrics-client-ca\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-sys\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.190999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.190990 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/12fa0c31-de73-4d43-8635-82e60062a477-root\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.191156 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.191076 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.191363 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.191337 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-textfile\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.191483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.191458 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0c31-de73-4d43-8635-82e60062a477-metrics-client-ca\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.193053 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.193029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-tls\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.193494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.193467 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12fa0c31-de73-4d43-8635-82e60062a477-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.202349 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.202320 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nh7r\" (UniqueName: \"kubernetes.io/projected/12fa0c31-de73-4d43-8635-82e60062a477-kube-api-access-2nh7r\") pod \"node-exporter-tsvt7\" (UID: \"12fa0c31-de73-4d43-8635-82e60062a477\") " pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.308846 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.308805 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tsvt7" Jul 24 15:10:40.342609 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.342527 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:40.534655 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.534600 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:40.693283 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.693175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:40.831664 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:40.831632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:41.071722 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:41.071676 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:41.357686 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:41.357604 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:41.468828 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:41.468788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:41.835864 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:41.835829 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fa0c31_de73_4d43_8635_82e60062a477.slice/crio-f2581c61f13567fc5f499bc5ca0e3ce3c5b4f4711583b03cb1779af63c5b7fb8 WatchSource:0}: Error finding container f2581c61f13567fc5f499bc5ca0e3ce3c5b4f4711583b03cb1779af63c5b7fb8: Status 404 returned error can't find the container with id f2581c61f13567fc5f499bc5ca0e3ce3c5b4f4711583b03cb1779af63c5b7fb8 Jul 24 15:10:42.023632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:42.023596 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcm7r" event={"ID":"0b5ca429-11b2-4d36-9222-8f3f71530f7d","Type":"ContainerStarted","Data":"25b54f3add69bdc4c7ce209d9d20d70941597c69f905c74b1169c7697310e9fa"} Jul 24 15:10:42.024679 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:42.024657 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsvt7" event={"ID":"12fa0c31-de73-4d43-8635-82e60062a477","Type":"ContainerStarted","Data":"f2581c61f13567fc5f499bc5ca0e3ce3c5b4f4711583b03cb1779af63c5b7fb8"} Jul 24 15:10:42.049231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:42.049165 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jcm7r" podStartSLOduration=34.492362203 podStartE2EDuration="46.04914677s" totalImagesPullingTime="11.556784567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:09:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:42.048794354 +0000 UTC m=+156.135446292" watchObservedRunningTime="2026-07-24 15:10:42.04914677 +0000 UTC m=+156.135798683" Jul 24 15:10:42.943557 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:42.943525 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:43.030126 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:43.030095 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsvt7" event={"ID":"12fa0c31-de73-4d43-8635-82e60062a477","Type":"ContainerStarted","Data":"16bc1ac65ed082fcdb6149bab57c6bcbd5f993324cf1c4ee653dccd8d834b0ee"} Jul 24 15:10:43.062222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:43.062179 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:43.421112 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:43.421072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:44.034689 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.034647 2563 generic.go:356] "Generic (PLEG): container finished" podID="12fa0c31-de73-4d43-8635-82e60062a477" containerID="16bc1ac65ed082fcdb6149bab57c6bcbd5f993324cf1c4ee653dccd8d834b0ee" exitCode=0 Jul 24 15:10:44.035091 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.034741 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsvt7" event={"ID":"12fa0c31-de73-4d43-8635-82e60062a477","Type":"ContainerDied","Data":"16bc1ac65ed082fcdb6149bab57c6bcbd5f993324cf1c4ee653dccd8d834b0ee"} Jul 24 15:10:44.127840 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.127804 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:44.274358 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.274321 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:44.305543 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:44.305448 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" podUID="326402ae-916b-475a-842b-f8e2543c3b02" Jul 24 15:10:44.313636 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:10:44.313601 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-l9rg4" podUID="f157fb48-58c1-4f43-bc15-b8dd74dfcc02" Jul 24 15:10:44.376339 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.376303 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x"] Jul 24 15:10:44.379492 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.379473 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.382066 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.382038 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 15:10:44.382286 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.382269 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7hdkh\"" Jul 24 15:10:44.399873 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.399834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:44.400921 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.400896 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x"] Jul 24 15:10:44.527889 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.527845 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7pf\" (UniqueName: \"kubernetes.io/projected/94c14b7a-93f1-45ab-a38b-dc241af1d481-kube-api-access-gd7pf\") pod \"monitoring-plugin-68b7fc9574-cx28x\" (UID: \"94c14b7a-93f1-45ab-a38b-dc241af1d481\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.528083 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.527928 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94c14b7a-93f1-45ab-a38b-dc241af1d481-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cx28x\" (UID: \"94c14b7a-93f1-45ab-a38b-dc241af1d481\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.629448 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.629348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7pf\" (UniqueName: \"kubernetes.io/projected/94c14b7a-93f1-45ab-a38b-dc241af1d481-kube-api-access-gd7pf\") pod \"monitoring-plugin-68b7fc9574-cx28x\" (UID: \"94c14b7a-93f1-45ab-a38b-dc241af1d481\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.629448 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.629419 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94c14b7a-93f1-45ab-a38b-dc241af1d481-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cx28x\" (UID: \"94c14b7a-93f1-45ab-a38b-dc241af1d481\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.631941 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.631913 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/94c14b7a-93f1-45ab-a38b-dc241af1d481-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-cx28x\" (UID: \"94c14b7a-93f1-45ab-a38b-dc241af1d481\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.640808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.640778 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7pf\" (UniqueName: \"kubernetes.io/projected/94c14b7a-93f1-45ab-a38b-dc241af1d481-kube-api-access-gd7pf\") pod \"monitoring-plugin-68b7fc9574-cx28x\" (UID: \"94c14b7a-93f1-45ab-a38b-dc241af1d481\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.688937 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.688894 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:44.695144 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.695110 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:44.821164 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:44.821016 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x"] Jul 24 15:10:44.824647 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:44.824609 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c14b7a_93f1_45ab_a38b_dc241af1d481.slice/crio-dfb0d0df22f7d3573f498f17a1701116b079e90e22bce1a785310a55478cc39a WatchSource:0}: Error finding container dfb0d0df22f7d3573f498f17a1701116b079e90e22bce1a785310a55478cc39a: Status 404 returned error can't find the container with id dfb0d0df22f7d3573f498f17a1701116b079e90e22bce1a785310a55478cc39a Jul 24 15:10:45.039628 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:45.039590 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" event={"ID":"94c14b7a-93f1-45ab-a38b-dc241af1d481","Type":"ContainerStarted","Data":"dfb0d0df22f7d3573f498f17a1701116b079e90e22bce1a785310a55478cc39a"} Jul 24 15:10:45.041593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:45.041559 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsvt7" event={"ID":"12fa0c31-de73-4d43-8635-82e60062a477","Type":"ContainerStarted","Data":"7d17066177936e6f53c815c4baf99df8a4ba41cafab02c762e2db94e07fbd754"} Jul 24 15:10:45.041746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:45.041610 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9rg4" Jul 24 15:10:45.041746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:45.041617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsvt7" event={"ID":"12fa0c31-de73-4d43-8635-82e60062a477","Type":"ContainerStarted","Data":"cba30838d507397f5a028a00035471817a3fa13aa0048f231017daedb725c54c"} Jul 24 15:10:45.041746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:45.041635 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:10:45.068893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:45.068843 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tsvt7" podStartSLOduration=4.896853287 podStartE2EDuration="6.068826195s" totalImagesPullingTime="1.104269661s" totalInitContainerRuntime="67.703247ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:45.067824163 +0000 UTC m=+159.154476081" watchObservedRunningTime="2026-07-24 15:10:45.068826195 +0000 UTC m=+159.155478107" Jul 24 15:10:46.200510 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:46.200454 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:47.051288 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:47.051233 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" event={"ID":"94c14b7a-93f1-45ab-a38b-dc241af1d481","Type":"ContainerStarted","Data":"79e6fc8e8a93ddd44811f61086484dbacbd41666ab10044377d4a50b1ff313f8"} Jul 24 15:10:47.051497 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:47.051472 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:47.055910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:47.055882 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" Jul 24 15:10:47.130785 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:47.130733 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-cx28x" podStartSLOduration=1.359637905 podStartE2EDuration="3.130717337s" totalImagesPullingTime="1.771079432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:47.130592126 +0000 UTC m=+161.217244031" watchObservedRunningTime="2026-07-24 15:10:47.130717337 +0000 UTC m=+161.217369228" Jul 24 15:10:49.170272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.169798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:10:49.173873 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.173473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"image-registry-6579c999df-lmfnm\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:10:49.245703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.245664 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p5plk\"" Jul 24 15:10:49.252900 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.252870 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:10:49.271260 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.271218 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:10:49.273863 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.273827 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f157fb48-58c1-4f43-bc15-b8dd74dfcc02-metrics-tls\") pod \"dns-default-l9rg4\" (UID: \"f157fb48-58c1-4f43-bc15-b8dd74dfcc02\") " pod="openshift-dns/dns-default-l9rg4" Jul 24 15:10:49.389700 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.389663 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6579c999df-lmfnm"] Jul 24 15:10:49.392883 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:49.392851 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326402ae_916b_475a_842b_f8e2543c3b02.slice/crio-45aac18288aa4f1bdd090ace027fb815cd0062a265116c44c8e79d0bad70a243 WatchSource:0}: Error finding container 45aac18288aa4f1bdd090ace027fb815cd0062a265116c44c8e79d0bad70a243: Status 404 returned error can't find the container with id 45aac18288aa4f1bdd090ace027fb815cd0062a265116c44c8e79d0bad70a243 Jul 24 15:10:49.545433 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.545400 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkfwv\"" Jul 24 15:10:49.553544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.553515 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9rg4" Jul 24 15:10:49.687229 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.687143 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9rg4"] Jul 24 15:10:49.690266 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:10:49.690229 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf157fb48_58c1_4f43_bc15_b8dd74dfcc02.slice/crio-29a7e8e5440eddcddf8af9205cff36f2ec92273bf7908817ca578845ef88c18a WatchSource:0}: Error finding container 29a7e8e5440eddcddf8af9205cff36f2ec92273bf7908817ca578845ef88c18a: Status 404 returned error can't find the container with id 29a7e8e5440eddcddf8af9205cff36f2ec92273bf7908817ca578845ef88c18a Jul 24 15:10:49.750802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.750756 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6579c999df-lmfnm"] Jul 24 15:10:49.848414 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.848370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:49.949848 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:49.949754 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:50.062651 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:50.062618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" event={"ID":"326402ae-916b-475a-842b-f8e2543c3b02","Type":"ContainerStarted","Data":"874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d"} Jul 24 15:10:50.062862 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:50.062670 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" event={"ID":"326402ae-916b-475a-842b-f8e2543c3b02","Type":"ContainerStarted","Data":"45aac18288aa4f1bdd090ace027fb815cd0062a265116c44c8e79d0bad70a243"} Jul 24 15:10:50.062862 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:50.062738 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:10:50.063804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:50.063774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9rg4" event={"ID":"f157fb48-58c1-4f43-bc15-b8dd74dfcc02","Type":"ContainerStarted","Data":"29a7e8e5440eddcddf8af9205cff36f2ec92273bf7908817ca578845ef88c18a"} Jul 24 15:10:50.086424 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:50.086372 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" podStartSLOduration=164.086354058 podStartE2EDuration="2m44.086354058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:50.08496144 +0000 UTC m=+164.171613353" watchObservedRunningTime="2026-07-24 15:10:50.086354058 +0000 UTC m=+164.173005970" Jul 24 15:10:52.072842 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:52.072808 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9rg4" event={"ID":"f157fb48-58c1-4f43-bc15-b8dd74dfcc02","Type":"ContainerStarted","Data":"56a291b00f710b6fd4dbff10f555ae8307d1397cfec69f52c8361a86539ed47c"} Jul 24 15:10:52.073338 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:52.072858 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9rg4" event={"ID":"f157fb48-58c1-4f43-bc15-b8dd74dfcc02","Type":"ContainerStarted","Data":"caa21441905db0b4a38fd7257603682b3b3bdd36febd311dd008e7a4b5fe1c36"} Jul 24 15:10:52.073338 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:52.072932 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l9rg4" Jul 24 15:10:52.098526 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:52.098467 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l9rg4" podStartSLOduration=129.177971941 podStartE2EDuration="2m11.098448309s" totalImagesPullingTime="1.920476368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:52.097482323 +0000 UTC m=+166.184134238" watchObservedRunningTime="2026-07-24 15:10:52.098448309 +0000 UTC m=+166.185100221" Jul 24 15:10:52.795836 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:52.795795 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:52.863420 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:52.863390 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:55.676601 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:55.676566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:55.771465 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:55.771398 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:57.240151 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:57.240120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:57.340138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:57.340103 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:58.016076 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:58.016039 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:10:58.119510 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:10:58.119477 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:02.079189 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:02.079110 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l9rg4" Jul 24 15:11:03.347087 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:03.347059 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9rg4_f157fb48-58c1-4f43-bc15-b8dd74dfcc02/dns/0.log" Jul 24 15:11:03.553100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:03.553068 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9rg4_f157fb48-58c1-4f43-bc15-b8dd74dfcc02/kube-rbac-proxy/0.log" Jul 24 15:11:04.148029 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:04.147997 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-656vw_40253b47-eaff-437d-a540-2526e5e4f40d/dns-node-resolver/0.log" Jul 24 15:11:05.347081 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:05.347045 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zw8rr_c3db899e-09ba-4b52-9b93-81c946f895e1/serve-healthcheck-canary/0.log" Jul 24 15:11:07.792000 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:07.791963 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:07.857601 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:07.857554 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:11.073385 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:11.073359 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:11:11.227290 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:11.227232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:11.296619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:11.296580 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:12.143796 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:12.143759 2563 generic.go:356] "Generic (PLEG): container finished" podID="8530a5e5-08c1-4d38-9d9c-429f9bf677c5" containerID="a18c1b96cef709920753e2e5ccce85fb81b8dcf2055a9de420c8147b98a7f086" exitCode=0 Jul 24 15:11:12.144223 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:12.143834 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" event={"ID":"8530a5e5-08c1-4d38-9d9c-429f9bf677c5","Type":"ContainerDied","Data":"a18c1b96cef709920753e2e5ccce85fb81b8dcf2055a9de420c8147b98a7f086"} Jul 24 15:11:12.144223 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:12.144147 2563 scope.go:117] "RemoveContainer" containerID="a18c1b96cef709920753e2e5ccce85fb81b8dcf2055a9de420c8147b98a7f086" Jul 24 15:11:13.150517 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:13.150480 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f" event={"ID":"8530a5e5-08c1-4d38-9d9c-429f9bf677c5","Type":"ContainerStarted","Data":"8a3100532bb52ee7bbbe423eb7b70a8a2034f9180b684d09a5196739b616a2b2"} Jul 24 15:11:16.085686 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.085638 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" podUID="326402ae-916b-475a-842b-f8e2543c3b02" containerName="registry" containerID="cri-o://874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d" gracePeriod=30 Jul 24 15:11:16.826947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.826919 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:11:16.878171 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878131 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-trusted-ca\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878171 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878184 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878207 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-installation-pull-secrets\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878260 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/326402ae-916b-475a-842b-f8e2543c3b02-ca-trust-extracted\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878401 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksdlc\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-kube-api-access-ksdlc\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878630 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878480 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-bound-sa-token\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878630 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878530 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-image-registry-private-configuration\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878630 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878596 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-registry-certificates\") pod \"326402ae-916b-475a-842b-f8e2543c3b02\" (UID: \"326402ae-916b-475a-842b-f8e2543c3b02\") " Jul 24 15:11:16.878784 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878696 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:11:16.878845 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.878809 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-trusted-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.879178 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.879130 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:11:16.881221 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.881187 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:11:16.881221 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.881200 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:11:16.881377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.881192 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-kube-api-access-ksdlc" (OuterVolumeSpecName: "kube-api-access-ksdlc") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "kube-api-access-ksdlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:11:16.881377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.881273 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:11:16.881377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.881302 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:11:16.886621 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.886593 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326402ae-916b-475a-842b-f8e2543c3b02-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "326402ae-916b-475a-842b-f8e2543c3b02" (UID: "326402ae-916b-475a-842b-f8e2543c3b02"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979407 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-installation-pull-secrets\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979450 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/326402ae-916b-475a-842b-f8e2543c3b02-ca-trust-extracted\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979460 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksdlc\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-kube-api-access-ksdlc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979468 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-bound-sa-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979479 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/326402ae-916b-475a-842b-f8e2543c3b02-image-registry-private-configuration\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979488 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/326402ae-916b-475a-842b-f8e2543c3b02-registry-certificates\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.979504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:16.979497 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/326402ae-916b-475a-842b-f8e2543c3b02-registry-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:11:17.171685 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.171649 2563 generic.go:356] "Generic (PLEG): container finished" podID="326402ae-916b-475a-842b-f8e2543c3b02" containerID="874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d" exitCode=0 Jul 24 15:11:17.172090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.171691 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" Jul 24 15:11:17.172090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.171733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" event={"ID":"326402ae-916b-475a-842b-f8e2543c3b02","Type":"ContainerDied","Data":"874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d"} Jul 24 15:11:17.172090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.171804 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6579c999df-lmfnm" event={"ID":"326402ae-916b-475a-842b-f8e2543c3b02","Type":"ContainerDied","Data":"45aac18288aa4f1bdd090ace027fb815cd0062a265116c44c8e79d0bad70a243"} Jul 24 15:11:17.172090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.171825 2563 scope.go:117] "RemoveContainer" containerID="874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d" Jul 24 15:11:17.181188 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.181171 2563 scope.go:117] "RemoveContainer" containerID="874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d" Jul 24 15:11:17.181568 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:11:17.181545 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d\": container with ID starting with 874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d not found: ID does not exist" containerID="874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d" Jul 24 15:11:17.181643 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.181575 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d"} err="failed to get container status \"874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d\": rpc error: code = NotFound desc = could not find container \"874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d\": container with ID starting with 874f73661daac91a9790c91e75a7263cf81946f3ed161c93fa3fd4ff356b8b2d not found: ID does not exist" Jul 24 15:11:17.198036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.198005 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6579c999df-lmfnm"] Jul 24 15:11:17.205593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:17.205565 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6579c999df-lmfnm"] Jul 24 15:11:18.436307 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:18.436269 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326402ae-916b-475a-842b-f8e2543c3b02" path="/var/lib/kubelet/pods/326402ae-916b-475a-842b-f8e2543c3b02/volumes" Jul 24 15:11:22.192172 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:22.192137 2563 generic.go:356] "Generic (PLEG): container finished" podID="8d65cb26-ec51-4d21-9839-2bc01a1f7d47" containerID="e4d804fa68d6e9578f2c5bd03ed24d46ee9cb32a4d1e4174802e55d425969e58" exitCode=0 Jul 24 15:11:22.192549 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:22.192214 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" event={"ID":"8d65cb26-ec51-4d21-9839-2bc01a1f7d47","Type":"ContainerDied","Data":"e4d804fa68d6e9578f2c5bd03ed24d46ee9cb32a4d1e4174802e55d425969e58"} Jul 24 15:11:22.192612 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:22.192551 2563 scope.go:117] "RemoveContainer" containerID="e4d804fa68d6e9578f2c5bd03ed24d46ee9cb32a4d1e4174802e55d425969e58" Jul 24 15:11:22.835738 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:22.835695 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:22.934307 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:22.934268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:23.197884 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:23.197797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-jw8gz" event={"ID":"8d65cb26-ec51-4d21-9839-2bc01a1f7d47","Type":"ContainerStarted","Data":"c9e2ccfce958277698c977fb535fbbe210dff7e687714d2133e3b95bfb8b74df"} Jul 24 15:11:32.380887 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:32.380844 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:32.481293 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:32.481224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:32.583662 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:32.583627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:11:32.683067 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:11:32.682976 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:01.552742 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:01.552686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:03.939688 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:03.939649 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:04.373495 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:04.373458 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:09.092008 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.091974 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-677cf8999f-g5rgl"] Jul 24 15:12:09.092442 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.092203 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326402ae-916b-475a-842b-f8e2543c3b02" containerName="registry" Jul 24 15:12:09.092442 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.092213 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="326402ae-916b-475a-842b-f8e2543c3b02" containerName="registry" Jul 24 15:12:09.092442 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.092285 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="326402ae-916b-475a-842b-f8e2543c3b02" containerName="registry" Jul 24 15:12:09.099612 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.099576 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.103042 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.103017 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 15:12:09.104074 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.104055 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 15:12:09.104192 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.104113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 15:12:09.104268 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.104215 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-spdfl\"" Jul 24 15:12:09.104326 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.104282 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 15:12:09.104624 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.104608 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 15:12:09.109030 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.109002 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 15:12:09.115591 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.115561 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-677cf8999f-g5rgl"] Jul 24 15:12:09.156091 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-federate-client-tls\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156091 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156096 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-secret-telemeter-client\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156198 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-serving-certs-ca-bundle\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156288 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-metrics-client-ca\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156327 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-telemeter-client-tls\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.156402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.156358 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6dz\" (UniqueName: \"kubernetes.io/projected/32912013-4bb2-466e-ba52-9e704f63b6f1-kube-api-access-qz6dz\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257365 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6dz\" (UniqueName: \"kubernetes.io/projected/32912013-4bb2-466e-ba52-9e704f63b6f1-kube-api-access-qz6dz\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257382 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-federate-client-tls\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257439 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-secret-telemeter-client\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257678 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-serving-certs-ca-bundle\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257678 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257635 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-metrics-client-ca\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.257794 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.257691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-telemeter-client-tls\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.258362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.258333 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-serving-certs-ca-bundle\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.258579 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.258554 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.258641 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.258622 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32912013-4bb2-466e-ba52-9e704f63b6f1-metrics-client-ca\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.260226 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.260199 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-secret-telemeter-client\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.260339 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.260231 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-telemeter-client-tls\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.260427 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.260412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.260494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.260474 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/32912013-4bb2-466e-ba52-9e704f63b6f1-federate-client-tls\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.268558 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.268532 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6dz\" (UniqueName: \"kubernetes.io/projected/32912013-4bb2-466e-ba52-9e704f63b6f1-kube-api-access-qz6dz\") pod \"telemeter-client-677cf8999f-g5rgl\" (UID: \"32912013-4bb2-466e-ba52-9e704f63b6f1\") " pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.409712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.409610 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" Jul 24 15:12:09.572972 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:09.572938 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-677cf8999f-g5rgl"] Jul 24 15:12:09.576634 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:12:09.576605 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32912013_4bb2_466e_ba52_9e704f63b6f1.slice/crio-2f6c8f12f67bd6dfaf5f1f1b95bd9d9bac1285fd200d8fd8c86094cb34951363 WatchSource:0}: Error finding container 2f6c8f12f67bd6dfaf5f1f1b95bd9d9bac1285fd200d8fd8c86094cb34951363: Status 404 returned error can't find the container with id 2f6c8f12f67bd6dfaf5f1f1b95bd9d9bac1285fd200d8fd8c86094cb34951363 Jul 24 15:12:10.364790 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:10.364749 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" event={"ID":"32912013-4bb2-466e-ba52-9e704f63b6f1","Type":"ContainerStarted","Data":"2f6c8f12f67bd6dfaf5f1f1b95bd9d9bac1285fd200d8fd8c86094cb34951363"} Jul 24 15:12:12.373039 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:12.372996 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" event={"ID":"32912013-4bb2-466e-ba52-9e704f63b6f1","Type":"ContainerStarted","Data":"1a49a7f879643b01ee55b906049ac4f5cda6ab2a582e43a63d07e8667c41416c"} Jul 24 15:12:13.378759 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:13.378713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" event={"ID":"32912013-4bb2-466e-ba52-9e704f63b6f1","Type":"ContainerStarted","Data":"90093207bdef36d515c16d4f99dbc6502a864537305087c44828147dfd86ba6b"} Jul 24 15:12:13.379139 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:13.378823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" event={"ID":"32912013-4bb2-466e-ba52-9e704f63b6f1","Type":"ContainerStarted","Data":"f818091b837f583e47caf55d7d24b6d29ebc24bc573d84cbe4bbab1258c3c52d"} Jul 24 15:12:13.410020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:13.409490 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-677cf8999f-g5rgl" podStartSLOduration=0.9064628 podStartE2EDuration="4.409469269s" totalImagesPullingTime="3.503006469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:12:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:12:13.408203924 +0000 UTC m=+247.494855850" watchObservedRunningTime="2026-07-24 15:12:13.409469269 +0000 UTC m=+247.496121185" Jul 24 15:12:14.389095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:14.389059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:14.489921 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:14.489869 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:19.263859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:19.263822 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:19.349493 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:19.349422 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:24.410036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:24.409982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:24.567552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:24.567507 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:25.000730 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:25.000690 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:25.097061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:25.097026 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:31.091302 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:31.091263 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:31.191020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:31.190971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:32.922150 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:32.922108 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:32.996827 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:32.996783 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:35.533194 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:35.533137 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:35.631822 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:35.631791 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:35.734287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:35.734221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:37.803090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:37.803051 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:37.915565 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:37.915531 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:38.009633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:38.009583 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:45.062830 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:45.062790 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:12:56.418282 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:12:56.418228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:06.312370 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:06.312337 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:13:06.314990 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:06.314963 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:13:06.325514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:06.325487 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:13:20.733666 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:20.733632 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:20.840525 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:20.840481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:21.856300 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.856262 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb"] Jul 24 15:13:21.859426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.859404 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.862805 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.862783 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:13:21.863305 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.863287 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:13:21.865438 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.865419 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z2b9l\"" Jul 24 15:13:21.888864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.888825 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.889039 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.888880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54x5\" (UniqueName: \"kubernetes.io/projected/7152844e-9170-4429-8f05-5654210a8bd1-kube-api-access-h54x5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.889039 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.888947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.892021 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.891990 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb"] Jul 24 15:13:21.990325 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.990277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.990505 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.990342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54x5\" (UniqueName: \"kubernetes.io/projected/7152844e-9170-4429-8f05-5654210a8bd1-kube-api-access-h54x5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.990505 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.990369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.990643 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.990614 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:21.990716 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:21.990657 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:22.000297 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:22.000267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54x5\" (UniqueName: \"kubernetes.io/projected/7152844e-9170-4429-8f05-5654210a8bd1-kube-api-access-h54x5\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:22.169133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:22.169040 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:22.312573 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:22.312540 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb"] Jul 24 15:13:22.315886 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:13:22.315859 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7152844e_9170_4429_8f05_5654210a8bd1.slice/crio-32b9e4f8929d6c8388a070a2f6b398f56045b0bdf72de4c47fff62a9964f6b55 WatchSource:0}: Error finding container 32b9e4f8929d6c8388a070a2f6b398f56045b0bdf72de4c47fff62a9964f6b55: Status 404 returned error can't find the container with id 32b9e4f8929d6c8388a070a2f6b398f56045b0bdf72de4c47fff62a9964f6b55 Jul 24 15:13:22.317776 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:22.317758 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:13:22.621351 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:22.621317 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" event={"ID":"7152844e-9170-4429-8f05-5654210a8bd1","Type":"ContainerStarted","Data":"32b9e4f8929d6c8388a070a2f6b398f56045b0bdf72de4c47fff62a9964f6b55"} Jul 24 15:13:29.654314 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:29.654206 2563 generic.go:356] "Generic (PLEG): container finished" podID="7152844e-9170-4429-8f05-5654210a8bd1" containerID="3a0c70ad6f65cdfc718d9cbbd17e0dc998aa0c10559f3d7e7cf0fda9a21ee6d6" exitCode=0 Jul 24 15:13:29.654314 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:29.654292 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" event={"ID":"7152844e-9170-4429-8f05-5654210a8bd1","Type":"ContainerDied","Data":"3a0c70ad6f65cdfc718d9cbbd17e0dc998aa0c10559f3d7e7cf0fda9a21ee6d6"} Jul 24 15:13:32.667145 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:32.667102 2563 generic.go:356] "Generic (PLEG): container finished" podID="7152844e-9170-4429-8f05-5654210a8bd1" containerID="b07a0be8bc2339ae5d0d532a200e7d0ecd054cf8e992960eecaace7560067e17" exitCode=0 Jul 24 15:13:32.667554 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:32.667194 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" event={"ID":"7152844e-9170-4429-8f05-5654210a8bd1","Type":"ContainerDied","Data":"b07a0be8bc2339ae5d0d532a200e7d0ecd054cf8e992960eecaace7560067e17"} Jul 24 15:13:40.699217 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:40.699180 2563 generic.go:356] "Generic (PLEG): container finished" podID="7152844e-9170-4429-8f05-5654210a8bd1" containerID="42e560b0ac36f8c0c663809b6b705be62c5167255654173317a8c0a2f850ba9d" exitCode=0 Jul 24 15:13:40.699636 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:40.699260 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" event={"ID":"7152844e-9170-4429-8f05-5654210a8bd1","Type":"ContainerDied","Data":"42e560b0ac36f8c0c663809b6b705be62c5167255654173317a8c0a2f850ba9d"} Jul 24 15:13:41.826366 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.826339 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:41.957880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.957787 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h54x5\" (UniqueName: \"kubernetes.io/projected/7152844e-9170-4429-8f05-5654210a8bd1-kube-api-access-h54x5\") pod \"7152844e-9170-4429-8f05-5654210a8bd1\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " Jul 24 15:13:41.958059 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.957887 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-util\") pod \"7152844e-9170-4429-8f05-5654210a8bd1\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " Jul 24 15:13:41.958059 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.957921 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-bundle\") pod \"7152844e-9170-4429-8f05-5654210a8bd1\" (UID: \"7152844e-9170-4429-8f05-5654210a8bd1\") " Jul 24 15:13:41.958604 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.958580 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-bundle" (OuterVolumeSpecName: "bundle") pod "7152844e-9170-4429-8f05-5654210a8bd1" (UID: "7152844e-9170-4429-8f05-5654210a8bd1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:13:41.959968 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.959938 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7152844e-9170-4429-8f05-5654210a8bd1-kube-api-access-h54x5" (OuterVolumeSpecName: "kube-api-access-h54x5") pod "7152844e-9170-4429-8f05-5654210a8bd1" (UID: "7152844e-9170-4429-8f05-5654210a8bd1"). InnerVolumeSpecName "kube-api-access-h54x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:13:41.962161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:41.962131 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-util" (OuterVolumeSpecName: "util") pod "7152844e-9170-4429-8f05-5654210a8bd1" (UID: "7152844e-9170-4429-8f05-5654210a8bd1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:13:42.058503 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:42.058452 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:13:42.058503 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:42.058499 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h54x5\" (UniqueName: \"kubernetes.io/projected/7152844e-9170-4429-8f05-5654210a8bd1-kube-api-access-h54x5\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:13:42.058503 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:42.058512 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7152844e-9170-4429-8f05-5654210a8bd1-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:13:42.708514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:42.708417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" event={"ID":"7152844e-9170-4429-8f05-5654210a8bd1","Type":"ContainerDied","Data":"32b9e4f8929d6c8388a070a2f6b398f56045b0bdf72de4c47fff62a9964f6b55"} Jul 24 15:13:42.708514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:42.708440 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dx9hb" Jul 24 15:13:42.708514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:42.708471 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b9e4f8929d6c8388a070a2f6b398f56045b0bdf72de4c47fff62a9964f6b55" Jul 24 15:13:48.138571 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:48.138536 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:48.481222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:48.481112 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:50.714276 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714223 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps"] Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714587 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="util" Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714605 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="util" Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714627 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="pull" Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714636 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="pull" Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714644 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="extract" Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714653 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="extract" Jul 24 15:13:50.714786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.714754 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7152844e-9170-4429-8f05-5654210a8bd1" containerName="extract" Jul 24 15:13:50.718524 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.718501 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:50.723557 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.723523 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 15:13:50.723703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.723653 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jc897\"" Jul 24 15:13:50.723703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.723653 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:13:50.767098 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.767064 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps"] Jul 24 15:13:50.827493 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.827448 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgjl\" (UniqueName: \"kubernetes.io/projected/7317d5b0-4ab9-4d38-b085-507e48307d13-kube-api-access-nwgjl\") pod \"cert-manager-operator-controller-manager-55c444bf86-vflps\" (UID: \"7317d5b0-4ab9-4d38-b085-507e48307d13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:50.827676 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.827531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7317d5b0-4ab9-4d38-b085-507e48307d13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vflps\" (UID: \"7317d5b0-4ab9-4d38-b085-507e48307d13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:50.928110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.928076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7317d5b0-4ab9-4d38-b085-507e48307d13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vflps\" (UID: \"7317d5b0-4ab9-4d38-b085-507e48307d13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:50.928238 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.928129 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgjl\" (UniqueName: \"kubernetes.io/projected/7317d5b0-4ab9-4d38-b085-507e48307d13-kube-api-access-nwgjl\") pod \"cert-manager-operator-controller-manager-55c444bf86-vflps\" (UID: \"7317d5b0-4ab9-4d38-b085-507e48307d13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:50.928490 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.928469 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7317d5b0-4ab9-4d38-b085-507e48307d13-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vflps\" (UID: \"7317d5b0-4ab9-4d38-b085-507e48307d13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:50.941503 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:50.941472 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgjl\" (UniqueName: \"kubernetes.io/projected/7317d5b0-4ab9-4d38-b085-507e48307d13-kube-api-access-nwgjl\") pod \"cert-manager-operator-controller-manager-55c444bf86-vflps\" (UID: \"7317d5b0-4ab9-4d38-b085-507e48307d13\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:51.027321 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:51.027289 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" Jul 24 15:13:51.240262 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:51.240197 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps"] Jul 24 15:13:51.244894 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:13:51.244861 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7317d5b0_4ab9_4d38_b085_507e48307d13.slice/crio-3a2e70aef6792e0e3553ae83e480ea94ebf575082f54a8f5e84e22d2752c14b9 WatchSource:0}: Error finding container 3a2e70aef6792e0e3553ae83e480ea94ebf575082f54a8f5e84e22d2752c14b9: Status 404 returned error can't find the container with id 3a2e70aef6792e0e3553ae83e480ea94ebf575082f54a8f5e84e22d2752c14b9 Jul 24 15:13:51.742522 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:51.742490 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" event={"ID":"7317d5b0-4ab9-4d38-b085-507e48307d13","Type":"ContainerStarted","Data":"3a2e70aef6792e0e3553ae83e480ea94ebf575082f54a8f5e84e22d2752c14b9"} Jul 24 15:13:52.389070 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:52.389027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:52.491990 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:52.491907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:53.753732 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:53.753695 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" event={"ID":"7317d5b0-4ab9-4d38-b085-507e48307d13","Type":"ContainerStarted","Data":"8e200d4634dda7d67260b27ea4abd8ec9df340108438d40fa3dd9a604d157951"} Jul 24 15:13:53.778144 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:53.778090 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vflps" podStartSLOduration=2.2413001 podStartE2EDuration="3.778071417s" totalImagesPullingTime="1.536771317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:13:53.776165749 +0000 UTC m=+347.862817662" watchObservedRunningTime="2026-07-24 15:13:53.778071417 +0000 UTC m=+347.864723330" Jul 24 15:13:54.226768 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.226728 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:54.506014 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.505926 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:54.567842 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.567809 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k"] Jul 24 15:13:54.569956 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.569940 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.582875 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.582846 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:13:54.583031 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.582948 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z2b9l\"" Jul 24 15:13:54.585355 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.585335 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:13:54.625768 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.625730 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k"] Jul 24 15:13:54.656910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.656874 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.657095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.656931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsrp\" (UniqueName: \"kubernetes.io/projected/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-kube-api-access-jvsrp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.657095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.657026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.702450 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.702414 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:54.757566 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.757480 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.757566 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.757560 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.758080 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.757596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsrp\" (UniqueName: \"kubernetes.io/projected/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-kube-api-access-jvsrp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.758080 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.757872 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.758080 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.757943 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.782366 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.782333 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsrp\" (UniqueName: \"kubernetes.io/projected/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-kube-api-access-jvsrp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.879349 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.879308 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:13:54.897188 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:54.897148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:55.025440 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:55.025412 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k"] Jul 24 15:13:55.027830 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:13:55.027803 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb23ce5_15e4_45ea_8a27_85f44cc2e37e.slice/crio-1cd560cbfc314dd8c34bf1e2f9cbadc1d3889117c264288a28f6131b4ff6fe49 WatchSource:0}: Error finding container 1cd560cbfc314dd8c34bf1e2f9cbadc1d3889117c264288a28f6131b4ff6fe49: Status 404 returned error can't find the container with id 1cd560cbfc314dd8c34bf1e2f9cbadc1d3889117c264288a28f6131b4ff6fe49 Jul 24 15:13:55.763449 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:55.763405 2563 generic.go:356] "Generic (PLEG): container finished" podID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerID="decdd3e9bbdde4970c1fb5093db88c48eb74020ed9160b7810313b7cd0333583" exitCode=0 Jul 24 15:13:55.763856 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:55.763490 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" event={"ID":"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e","Type":"ContainerDied","Data":"decdd3e9bbdde4970c1fb5093db88c48eb74020ed9160b7810313b7cd0333583"} Jul 24 15:13:55.763856 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:55.763570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" event={"ID":"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e","Type":"ContainerStarted","Data":"1cd560cbfc314dd8c34bf1e2f9cbadc1d3889117c264288a28f6131b4ff6fe49"} Jul 24 15:13:57.020759 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:57.020718 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:57.182108 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:57.182069 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:57.774704 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:57.774544 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" event={"ID":"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e","Type":"ContainerStarted","Data":"f9c221263a168809cbd158442762164c955d2210704e899a4021a7db955b0dcf"} Jul 24 15:13:58.746544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:58.746452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:58.780489 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:58.780456 2563 generic.go:356] "Generic (PLEG): container finished" podID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerID="f9c221263a168809cbd158442762164c955d2210704e899a4021a7db955b0dcf" exitCode=0 Jul 24 15:13:58.780692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:58.780542 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" event={"ID":"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e","Type":"ContainerDied","Data":"f9c221263a168809cbd158442762164c955d2210704e899a4021a7db955b0dcf"} Jul 24 15:13:58.813983 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:58.813947 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:13:59.786330 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:59.786296 2563 generic.go:356] "Generic (PLEG): container finished" podID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerID="09359a796934010c66d79dda73f3054a027a41b292dfd892f9b00809bde1811c" exitCode=0 Jul 24 15:13:59.786763 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:13:59.786375 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" event={"ID":"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e","Type":"ContainerDied","Data":"09359a796934010c66d79dda73f3054a027a41b292dfd892f9b00809bde1811c"} Jul 24 15:14:00.916374 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:00.916345 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:14:01.005320 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.005278 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvsrp\" (UniqueName: \"kubernetes.io/projected/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-kube-api-access-jvsrp\") pod \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " Jul 24 15:14:01.005498 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.005348 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-bundle\") pod \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " Jul 24 15:14:01.005498 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.005375 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-util\") pod \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\" (UID: \"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e\") " Jul 24 15:14:01.005815 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.005780 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-bundle" (OuterVolumeSpecName: "bundle") pod "5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" (UID: "5eb23ce5-15e4-45ea-8a27-85f44cc2e37e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:14:01.007455 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.007429 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-kube-api-access-jvsrp" (OuterVolumeSpecName: "kube-api-access-jvsrp") pod "5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" (UID: "5eb23ce5-15e4-45ea-8a27-85f44cc2e37e"). InnerVolumeSpecName "kube-api-access-jvsrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:14:01.010387 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.010359 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-util" (OuterVolumeSpecName: "util") pod "5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" (UID: "5eb23ce5-15e4-45ea-8a27-85f44cc2e37e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:14:01.106468 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.106373 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:14:01.106468 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.106406 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:14:01.106468 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.106416 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsrp\" (UniqueName: \"kubernetes.io/projected/5eb23ce5-15e4-45ea-8a27-85f44cc2e37e-kube-api-access-jvsrp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:14:01.796384 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.796354 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" Jul 24 15:14:01.796384 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.796364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frx42k" event={"ID":"5eb23ce5-15e4-45ea-8a27-85f44cc2e37e","Type":"ContainerDied","Data":"1cd560cbfc314dd8c34bf1e2f9cbadc1d3889117c264288a28f6131b4ff6fe49"} Jul 24 15:14:01.796638 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:01.796404 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cd560cbfc314dd8c34bf1e2f9cbadc1d3889117c264288a28f6131b4ff6fe49" Jul 24 15:14:04.593861 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:04.593819 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:05.134975 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.134942 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vgb2r"] Jul 24 15:14:05.135213 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135201 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="pull" Jul 24 15:14:05.135272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135214 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="pull" Jul 24 15:14:05.135272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135234 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="extract" Jul 24 15:14:05.135272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135255 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="extract" Jul 24 15:14:05.135272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135265 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="util" Jul 24 15:14:05.135272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135270 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="util" Jul 24 15:14:05.135428 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.135329 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb23ce5-15e4-45ea-8a27-85f44cc2e37e" containerName="extract" Jul 24 15:14:05.138099 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.138079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.140480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.140452 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:14:05.140633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.140617 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ndt6z\"" Jul 24 15:14:05.141347 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.141330 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:14:05.154981 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.154953 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vgb2r"] Jul 24 15:14:05.238932 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.238886 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df-bound-sa-token\") pod \"cert-manager-55b4d954d5-vgb2r\" (UID: \"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df\") " pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.239121 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.238945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swrg9\" (UniqueName: \"kubernetes.io/projected/e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df-kube-api-access-swrg9\") pod \"cert-manager-55b4d954d5-vgb2r\" (UID: \"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df\") " pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.340182 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.340145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swrg9\" (UniqueName: \"kubernetes.io/projected/e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df-kube-api-access-swrg9\") pod \"cert-manager-55b4d954d5-vgb2r\" (UID: \"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df\") " pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.340367 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.340214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df-bound-sa-token\") pod \"cert-manager-55b4d954d5-vgb2r\" (UID: \"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df\") " pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.349579 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.349538 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df-bound-sa-token\") pod \"cert-manager-55b4d954d5-vgb2r\" (UID: \"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df\") " pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.349750 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.349659 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swrg9\" (UniqueName: \"kubernetes.io/projected/e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df-kube-api-access-swrg9\") pod \"cert-manager-55b4d954d5-vgb2r\" (UID: \"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df\") " pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.447774 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.447685 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vgb2r" Jul 24 15:14:05.578543 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.578507 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:05.594238 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.594202 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vgb2r"] Jul 24 15:14:05.597974 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:05.597944 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1aa6d43_7dd8_4ba2_bf83_38db9f9cc1df.slice/crio-ddaee4b53fcae0e03391a05eaff48d7903cea6f858eb36aeb8dfac45abe5e2e6 WatchSource:0}: Error finding container ddaee4b53fcae0e03391a05eaff48d7903cea6f858eb36aeb8dfac45abe5e2e6: Status 404 returned error can't find the container with id ddaee4b53fcae0e03391a05eaff48d7903cea6f858eb36aeb8dfac45abe5e2e6 Jul 24 15:14:05.664308 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.664267 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:05.813980 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:05.813947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vgb2r" event={"ID":"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df","Type":"ContainerStarted","Data":"ddaee4b53fcae0e03391a05eaff48d7903cea6f858eb36aeb8dfac45abe5e2e6"} Jul 24 15:14:06.725687 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.725647 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4"] Jul 24 15:14:06.729046 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.729022 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:06.737407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.737378 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:14:06.738171 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.738146 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vbvv8\"" Jul 24 15:14:06.738302 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.738146 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:14:06.770802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.770765 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4"] Jul 24 15:14:06.852672 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.852625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7svr\" (UniqueName: \"kubernetes.io/projected/327c309a-2737-44ea-8002-06e8a409114a-kube-api-access-k7svr\") pod \"openshift-lws-operator-fd8ccff4c-5mxs4\" (UID: \"327c309a-2737-44ea-8002-06e8a409114a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:06.852867 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.852698 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c309a-2737-44ea-8002-06e8a409114a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5mxs4\" (UID: \"327c309a-2737-44ea-8002-06e8a409114a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:06.953361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.953321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7svr\" (UniqueName: \"kubernetes.io/projected/327c309a-2737-44ea-8002-06e8a409114a-kube-api-access-k7svr\") pod \"openshift-lws-operator-fd8ccff4c-5mxs4\" (UID: \"327c309a-2737-44ea-8002-06e8a409114a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:06.953605 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.953384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c309a-2737-44ea-8002-06e8a409114a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5mxs4\" (UID: \"327c309a-2737-44ea-8002-06e8a409114a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:06.953813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.953782 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/327c309a-2737-44ea-8002-06e8a409114a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-5mxs4\" (UID: \"327c309a-2737-44ea-8002-06e8a409114a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:06.964748 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:06.964715 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7svr\" (UniqueName: \"kubernetes.io/projected/327c309a-2737-44ea-8002-06e8a409114a-kube-api-access-k7svr\") pod \"openshift-lws-operator-fd8ccff4c-5mxs4\" (UID: \"327c309a-2737-44ea-8002-06e8a409114a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:07.039084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:07.039045 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" Jul 24 15:14:07.189496 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:07.189461 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4"] Jul 24 15:14:08.060009 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:08.059970 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod327c309a_2737_44ea_8002_06e8a409114a.slice/crio-22fe49678356ab4677f1fa90dd53d964f355f6dc9396c0c92f352c606ce40301 WatchSource:0}: Error finding container 22fe49678356ab4677f1fa90dd53d964f355f6dc9396c0c92f352c606ce40301: Status 404 returned error can't find the container with id 22fe49678356ab4677f1fa90dd53d964f355f6dc9396c0c92f352c606ce40301 Jul 24 15:14:08.830948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:08.830828 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" event={"ID":"327c309a-2737-44ea-8002-06e8a409114a","Type":"ContainerStarted","Data":"22fe49678356ab4677f1fa90dd53d964f355f6dc9396c0c92f352c606ce40301"} Jul 24 15:14:08.832633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:08.832603 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vgb2r" event={"ID":"e1aa6d43-7dd8-4ba2-bf83-38db9f9cc1df","Type":"ContainerStarted","Data":"f673bd3a77e5b9cb22114d7996ac4e9e1ae68df5421f7a9d94d9ec507fcdbc45"} Jul 24 15:14:08.853469 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:08.853262 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vgb2r" podStartSLOduration=1.317347397 podStartE2EDuration="3.853218277s" totalImagesPullingTime="2.53587088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:08.85157964 +0000 UTC m=+362.938231557" watchObservedRunningTime="2026-07-24 15:14:08.853218277 +0000 UTC m=+362.939870192" Jul 24 15:14:09.102929 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:09.102841 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:09.204689 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:09.204627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:10.842706 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:10.842644 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" event={"ID":"327c309a-2737-44ea-8002-06e8a409114a","Type":"ContainerStarted","Data":"66c74a7e454fff7ea8f35c3b49bd8af89360aeddcef46fe6615e4c44fcf01057"} Jul 24 15:14:10.859767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:10.859706 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-5mxs4" podStartSLOduration=2.673430497 podStartE2EDuration="4.859680505s" totalImagesPullingTime="2.186250008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:10.859551272 +0000 UTC m=+364.946203185" watchObservedRunningTime="2026-07-24 15:14:10.859680505 +0000 UTC m=+364.946332421" Jul 24 15:14:11.653791 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:11.653750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:11.754193 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:11.754155 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:13.165875 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.165838 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8"] Jul 24 15:14:13.168150 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.168131 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.170639 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.170603 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:14:13.171460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.171438 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z2b9l\"" Jul 24 15:14:13.171586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.171438 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:14:13.179944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.179913 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8"] Jul 24 15:14:13.304855 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.304812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmq2\" (UniqueName: \"kubernetes.io/projected/83ccf5a0-475d-467e-aeb9-3ff5718e2801-kube-api-access-4tmq2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.304855 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.304862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.305080 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.304891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.405439 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.405397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.405639 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.405471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.405639 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.405598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmq2\" (UniqueName: \"kubernetes.io/projected/83ccf5a0-475d-467e-aeb9-3ff5718e2801-kube-api-access-4tmq2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.405826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.405806 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.405898 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.405875 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.414135 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.414092 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmq2\" (UniqueName: \"kubernetes.io/projected/83ccf5a0-475d-467e-aeb9-3ff5718e2801-kube-api-access-4tmq2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.449682 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.449607 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:13.478178 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.478139 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:13.614825 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.614759 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8"] Jul 24 15:14:13.617892 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:13.617863 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ccf5a0_475d_467e_aeb9_3ff5718e2801.slice/crio-dbdb870ec133bc68bd6c92022ecbaf0b404c19351646b9116088b059594fa895 WatchSource:0}: Error finding container dbdb870ec133bc68bd6c92022ecbaf0b404c19351646b9116088b059594fa895: Status 404 returned error can't find the container with id dbdb870ec133bc68bd6c92022ecbaf0b404c19351646b9116088b059594fa895 Jul 24 15:14:13.855858 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.855822 2563 generic.go:356] "Generic (PLEG): container finished" podID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerID="607ba8d6b123f25b0785eff149a5cf2d4b6bfcf841beadddb3fb4c2baaeb773d" exitCode=0 Jul 24 15:14:13.856026 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.855864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" event={"ID":"83ccf5a0-475d-467e-aeb9-3ff5718e2801","Type":"ContainerDied","Data":"607ba8d6b123f25b0785eff149a5cf2d4b6bfcf841beadddb3fb4c2baaeb773d"} Jul 24 15:14:13.856026 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:13.855905 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" event={"ID":"83ccf5a0-475d-467e-aeb9-3ff5718e2801","Type":"ContainerStarted","Data":"dbdb870ec133bc68bd6c92022ecbaf0b404c19351646b9116088b059594fa895"} Jul 24 15:14:14.864714 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:14.864669 2563 generic.go:356] "Generic (PLEG): container finished" podID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerID="8699e841a8000ecea2c7a00b9cbf31db144e46be3f7867d7edc24d939f141e28" exitCode=0 Jul 24 15:14:14.865118 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:14.864762 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" event={"ID":"83ccf5a0-475d-467e-aeb9-3ff5718e2801","Type":"ContainerDied","Data":"8699e841a8000ecea2c7a00b9cbf31db144e46be3f7867d7edc24d939f141e28"} Jul 24 15:14:15.872544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:15.872506 2563 generic.go:356] "Generic (PLEG): container finished" podID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerID="ca55795b66f12caa2311e79236f882878653128d4fdcaac588d7f61ec9430c00" exitCode=0 Jul 24 15:14:15.873011 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:15.872583 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" event={"ID":"83ccf5a0-475d-467e-aeb9-3ff5718e2801","Type":"ContainerDied","Data":"ca55795b66f12caa2311e79236f882878653128d4fdcaac588d7f61ec9430c00"} Jul 24 15:14:16.999529 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:16.999502 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:17.031539 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.031503 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-bundle\") pod \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " Jul 24 15:14:17.031782 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.031552 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-util\") pod \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " Jul 24 15:14:17.032406 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.032371 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-bundle" (OuterVolumeSpecName: "bundle") pod "83ccf5a0-475d-467e-aeb9-3ff5718e2801" (UID: "83ccf5a0-475d-467e-aeb9-3ff5718e2801"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:14:17.037881 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.037839 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-util" (OuterVolumeSpecName: "util") pod "83ccf5a0-475d-467e-aeb9-3ff5718e2801" (UID: "83ccf5a0-475d-467e-aeb9-3ff5718e2801"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:14:17.132704 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.132612 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tmq2\" (UniqueName: \"kubernetes.io/projected/83ccf5a0-475d-467e-aeb9-3ff5718e2801-kube-api-access-4tmq2\") pod \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\" (UID: \"83ccf5a0-475d-467e-aeb9-3ff5718e2801\") " Jul 24 15:14:17.132853 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.132786 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:14:17.132853 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.132797 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83ccf5a0-475d-467e-aeb9-3ff5718e2801-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:14:17.134643 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.134599 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ccf5a0-475d-467e-aeb9-3ff5718e2801-kube-api-access-4tmq2" (OuterVolumeSpecName: "kube-api-access-4tmq2") pod "83ccf5a0-475d-467e-aeb9-3ff5718e2801" (UID: "83ccf5a0-475d-467e-aeb9-3ff5718e2801"). InnerVolumeSpecName "kube-api-access-4tmq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:14:17.233098 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.233058 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmq2\" (UniqueName: \"kubernetes.io/projected/83ccf5a0-475d-467e-aeb9-3ff5718e2801-kube-api-access-4tmq2\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:14:17.881925 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.881892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" event={"ID":"83ccf5a0-475d-467e-aeb9-3ff5718e2801","Type":"ContainerDied","Data":"dbdb870ec133bc68bd6c92022ecbaf0b404c19351646b9116088b059594fa895"} Jul 24 15:14:17.882098 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.881943 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbdb870ec133bc68bd6c92022ecbaf0b404c19351646b9116088b059594fa895" Jul 24 15:14:17.882098 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.881900 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afjxlm8" Jul 24 15:14:17.964856 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:17.964804 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:18.155339 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:18.151861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:25.870788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:25.870733 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:27.422259 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422202 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx"] Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422546 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="pull" Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422565 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="pull" Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422582 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="util" Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422591 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="util" Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422612 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="extract" Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422621 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="extract" Jul 24 15:14:27.422752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.422725 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83ccf5a0-475d-467e-aeb9-3ff5718e2801" containerName="extract" Jul 24 15:14:27.425683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.425660 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.428436 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.428413 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:14:27.428572 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.428469 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:14:27.428572 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.428494 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vjk5s\"" Jul 24 15:14:27.428758 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.428740 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:14:27.429304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.429290 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:14:27.454045 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.453979 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx"] Jul 24 15:14:27.504295 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.504227 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95sr\" (UniqueName: \"kubernetes.io/projected/3ac15baf-c265-4ea2-9277-3fa87ba8a209-kube-api-access-s95sr\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.504449 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.504306 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ac15baf-c265-4ea2-9277-3fa87ba8a209-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.504449 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.504326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ac15baf-c265-4ea2-9277-3fa87ba8a209-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.504449 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.504382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ac15baf-c265-4ea2-9277-3fa87ba8a209-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.504449 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.504426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ac15baf-c265-4ea2-9277-3fa87ba8a209-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.604854 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.604821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95sr\" (UniqueName: \"kubernetes.io/projected/3ac15baf-c265-4ea2-9277-3fa87ba8a209-kube-api-access-s95sr\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.604854 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.604861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ac15baf-c265-4ea2-9277-3fa87ba8a209-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.605112 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.604880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ac15baf-c265-4ea2-9277-3fa87ba8a209-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.605112 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.605012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ac15baf-c265-4ea2-9277-3fa87ba8a209-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.605112 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.605062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ac15baf-c265-4ea2-9277-3fa87ba8a209-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.605332 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.605312 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ac15baf-c265-4ea2-9277-3fa87ba8a209-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.607409 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.607382 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ac15baf-c265-4ea2-9277-3fa87ba8a209-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.607527 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.607448 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ac15baf-c265-4ea2-9277-3fa87ba8a209-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.607636 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.607615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ac15baf-c265-4ea2-9277-3fa87ba8a209-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.619953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.619917 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95sr\" (UniqueName: \"kubernetes.io/projected/3ac15baf-c265-4ea2-9277-3fa87ba8a209-kube-api-access-s95sr\") pod \"opendatahub-operator-controller-manager-cff5b748-l8bjx\" (UID: \"3ac15baf-c265-4ea2-9277-3fa87ba8a209\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.640807 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.640769 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:27.736022 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.735932 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:27.740348 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.740313 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:27.851314 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.851284 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:27.881988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.881959 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx"] Jul 24 15:14:27.886919 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:27.886887 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac15baf_c265_4ea2_9277_3fa87ba8a209.slice/crio-676ae8841af8ecaf9a1a4b10eaaaf361b5ed0d2bf7dab34886513043c7715d24 WatchSource:0}: Error finding container 676ae8841af8ecaf9a1a4b10eaaaf361b5ed0d2bf7dab34886513043c7715d24: Status 404 returned error can't find the container with id 676ae8841af8ecaf9a1a4b10eaaaf361b5ed0d2bf7dab34886513043c7715d24 Jul 24 15:14:27.927911 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.927858 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" event={"ID":"3ac15baf-c265-4ea2-9277-3fa87ba8a209","Type":"ContainerStarted","Data":"676ae8841af8ecaf9a1a4b10eaaaf361b5ed0d2bf7dab34886513043c7715d24"} Jul 24 15:14:27.959238 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:27.959198 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:28.064101 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:28.064061 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:31.947465 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:31.947426 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" event={"ID":"3ac15baf-c265-4ea2-9277-3fa87ba8a209","Type":"ContainerStarted","Data":"4262326541d3eb41cb97b01c02a648fb852c6bb8b98db3b56a7646ed876e325a"} Jul 24 15:14:32.953633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:32.953598 2563 generic.go:356] "Generic (PLEG): container finished" podID="3ac15baf-c265-4ea2-9277-3fa87ba8a209" containerID="4262326541d3eb41cb97b01c02a648fb852c6bb8b98db3b56a7646ed876e325a" exitCode=0 Jul 24 15:14:32.954030 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:32.953664 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" event={"ID":"3ac15baf-c265-4ea2-9277-3fa87ba8a209","Type":"ContainerDied","Data":"4262326541d3eb41cb97b01c02a648fb852c6bb8b98db3b56a7646ed876e325a"} Jul 24 15:14:33.960133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:33.960099 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" event={"ID":"3ac15baf-c265-4ea2-9277-3fa87ba8a209","Type":"ContainerStarted","Data":"dd7923e12d7f906efe6eb24c3ac72e75c3f3d6f59bd62c042dc2ac94a59e7ea6"} Jul 24 15:14:33.960545 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:33.960217 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:33.998704 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:33.998626 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" podStartSLOduration=2.368406909 podStartE2EDuration="6.99861118s" totalImagesPullingTime="3.587966729s" totalInitContainerRuntime="1.042237542s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:33.998060748 +0000 UTC m=+388.084712662" watchObservedRunningTime="2026-07-24 15:14:33.99861118 +0000 UTC m=+388.085263093" Jul 24 15:14:43.619765 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:43.619725 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:43.760189 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:43.760148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:44.967999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:44.967965 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-l8bjx" Jul 24 15:14:46.936140 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:46.936105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:47.223723 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.223641 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:47.935992 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.935953 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:47.947251 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.947204 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m"] Jul 24 15:14:47.955523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.955491 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:47.960717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.960685 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 15:14:47.960893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.960773 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 15:14:47.961035 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.961015 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:14:47.961094 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:47.961015 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:14:48.042936 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.042904 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m"] Jul 24 15:14:48.073412 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.073369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81080541-f392-46d6-a35d-0861c5f5aae2-tmp\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.073592 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.073446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7z9x\" (UniqueName: \"kubernetes.io/projected/81080541-f392-46d6-a35d-0861c5f5aae2-kube-api-access-c7z9x\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.073592 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.073485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.174451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.174410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81080541-f392-46d6-a35d-0861c5f5aae2-tmp\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.174637 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.174471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7z9x\" (UniqueName: \"kubernetes.io/projected/81080541-f392-46d6-a35d-0861c5f5aae2-kube-api-access-c7z9x\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.174637 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.174499 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.174637 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:48.174601 2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 24 15:14:48.174778 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:48.174673 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs podName:81080541-f392-46d6-a35d-0861c5f5aae2 nodeName:}" failed. No retries permitted until 2026-07-24 15:14:48.674655816 +0000 UTC m=+402.761307710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs") pod "kube-auth-proxy-587457bbc9-kwb8m" (UID: "81080541-f392-46d6-a35d-0861c5f5aae2") : secret "kube-auth-proxy-tls" not found Jul 24 15:14:48.176701 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.176671 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81080541-f392-46d6-a35d-0861c5f5aae2-tmp\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.214399 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.214327 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7z9x\" (UniqueName: \"kubernetes.io/projected/81080541-f392-46d6-a35d-0861c5f5aae2-kube-api-access-c7z9x\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.436646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.436616 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-56ggm"] Jul 24 15:14:48.439893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.439874 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:48.446555 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.446529 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 15:14:48.447393 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.447372 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-clwdg\"" Jul 24 15:14:48.477029 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.476947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwts\" (UniqueName: \"kubernetes.io/projected/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-kube-api-access-dfwts\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:48.477029 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.477018 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-cert\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:48.483334 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.483303 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-56ggm"] Jul 24 15:14:48.570996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.570958 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:48.578033 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.578002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwts\" (UniqueName: \"kubernetes.io/projected/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-kube-api-access-dfwts\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:48.578165 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.578067 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-cert\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:48.578165 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:48.578159 2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 15:14:48.578279 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:48.578208 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-cert podName:96e8ac06-690b-4c08-8c7b-b9047c4f8b59 nodeName:}" failed. No retries permitted until 2026-07-24 15:14:49.078195097 +0000 UTC m=+403.164846988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-cert") pod "odh-model-controller-7998c67dc4-56ggm" (UID: "96e8ac06-690b-4c08-8c7b-b9047c4f8b59") : secret "odh-model-controller-webhook-cert" not found Jul 24 15:14:48.630540 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.630499 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwts\" (UniqueName: \"kubernetes.io/projected/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-kube-api-access-dfwts\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:48.679295 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.679225 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:48.679469 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:48.679372 2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 24 15:14:48.679469 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:48.679439 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs podName:81080541-f392-46d6-a35d-0861c5f5aae2 nodeName:}" failed. No retries permitted until 2026-07-24 15:14:49.679421858 +0000 UTC m=+403.766073752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs") pod "kube-auth-proxy-587457bbc9-kwb8m" (UID: "81080541-f392-46d6-a35d-0861c5f5aae2") : secret "kube-auth-proxy-tls" not found Jul 24 15:14:48.946270 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:48.946221 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:49.081948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.081908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-cert\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:49.084450 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.084423 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96e8ac06-690b-4c08-8c7b-b9047c4f8b59-cert\") pod \"odh-model-controller-7998c67dc4-56ggm\" (UID: \"96e8ac06-690b-4c08-8c7b-b9047c4f8b59\") " pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:49.242781 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.242664 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:49.350863 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.350819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:49.520200 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.520166 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-56ggm"] Jul 24 15:14:49.523996 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:49.523958 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e8ac06_690b_4c08_8c7b_b9047c4f8b59.slice/crio-6370ff26ff9915cdb17400b5e1d8c1818c396a98b1b7082723937ecbec64a5d0 WatchSource:0}: Error finding container 6370ff26ff9915cdb17400b5e1d8c1818c396a98b1b7082723937ecbec64a5d0: Status 404 returned error can't find the container with id 6370ff26ff9915cdb17400b5e1d8c1818c396a98b1b7082723937ecbec64a5d0 Jul 24 15:14:49.687406 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.687361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:49.689831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.689801 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81080541-f392-46d6-a35d-0861c5f5aae2-tls-certs\") pod \"kube-auth-proxy-587457bbc9-kwb8m\" (UID: \"81080541-f392-46d6-a35d-0861c5f5aae2\") " pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:49.765060 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.765017 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" Jul 24 15:14:49.961589 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:49.961564 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m"] Jul 24 15:14:49.963892 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:49.963850 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81080541_f392_46d6_a35d_0861c5f5aae2.slice/crio-0f34ac433975b81fe1d10aa3d621ccbdd3591c8b8d1fff16dfdcf15921135f90 WatchSource:0}: Error finding container 0f34ac433975b81fe1d10aa3d621ccbdd3591c8b8d1fff16dfdcf15921135f90: Status 404 returned error can't find the container with id 0f34ac433975b81fe1d10aa3d621ccbdd3591c8b8d1fff16dfdcf15921135f90 Jul 24 15:14:50.031929 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.031823 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" event={"ID":"81080541-f392-46d6-a35d-0861c5f5aae2","Type":"ContainerStarted","Data":"0f34ac433975b81fe1d10aa3d621ccbdd3591c8b8d1fff16dfdcf15921135f90"} Jul 24 15:14:50.032966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.032944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" event={"ID":"96e8ac06-690b-4c08-8c7b-b9047c4f8b59","Type":"ContainerStarted","Data":"6370ff26ff9915cdb17400b5e1d8c1818c396a98b1b7082723937ecbec64a5d0"} Jul 24 15:14:50.265044 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.264999 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vrdjb"] Jul 24 15:14:50.268716 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.268676 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:50.292488 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.292411 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 15:14:50.329090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.329027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:50.353093 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.353051 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vrdjb"] Jul 24 15:14:50.393621 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.393450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtgs\" (UniqueName: \"kubernetes.io/projected/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-kube-api-access-rmtgs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:50.393621 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.393500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:50.494090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.494050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtgs\" (UniqueName: \"kubernetes.io/projected/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-kube-api-access-rmtgs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:50.494316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.494106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:50.494386 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:50.494309 2563 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 15:14:50.494441 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:14:50.494388 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-tls-certs podName:faa859a6-6bbb-4318-a8fb-21b81df0c6e7 nodeName:}" failed. No retries permitted until 2026-07-24 15:14:50.994364994 +0000 UTC m=+405.081016890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-tls-certs") pod "model-serving-api-54d89bd5fc-vrdjb" (UID: "faa859a6-6bbb-4318-a8fb-21b81df0c6e7") : secret "model-serving-api-tls" not found Jul 24 15:14:50.536615 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.536548 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtgs\" (UniqueName: \"kubernetes.io/projected/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-kube-api-access-rmtgs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:50.863343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.863291 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:50.999596 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:50.999561 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:51.007320 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.004398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa859a6-6bbb-4318-a8fb-21b81df0c6e7-tls-certs\") pod \"model-serving-api-54d89bd5fc-vrdjb\" (UID: \"faa859a6-6bbb-4318-a8fb-21b81df0c6e7\") " pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:51.183183 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.183095 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:51.891186 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.891150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:51.921383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.921338 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv"] Jul 24 15:14:51.924653 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.924631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:51.928119 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.928089 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 15:14:51.928273 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.928144 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 15:14:51.928273 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.928144 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-772jw\"" Jul 24 15:14:51.928273 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.928089 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 15:14:51.928870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.928850 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 15:14:51.955022 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:51.954990 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv"] Jul 24 15:14:52.007930 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.007878 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.008110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.007941 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.008110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.007981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.008110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.008019 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.008110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.008046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0edaa9a-80ff-44f3-b066-d8285350c8fb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.008110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.008075 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtr9\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-kube-api-access-vmtr9\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.008110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.008102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.060742 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.060701 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:52.109528 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.109733 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.109733 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0edaa9a-80ff-44f3-b066-d8285350c8fb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.109733 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtr9\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-kube-api-access-vmtr9\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.109915 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.109915 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.109915 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.109861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.110340 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.110225 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.112548 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.112525 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0edaa9a-80ff-44f3-b066-d8285350c8fb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.112698 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.112679 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.112748 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.112694 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.113097 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.113077 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.120828 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.120791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtr9\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-kube-api-access-vmtr9\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.121336 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.121313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zjrlv\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:52.236581 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:52.236486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:53.130611 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:53.130579 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv"] Jul 24 15:14:53.132660 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:53.132633 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0edaa9a_80ff_44f3_b066_d8285350c8fb.slice/crio-f9ba651468f7e7cbd3e6e79c733d6046f88f556f4c03accc9e71040e4d0889b7 WatchSource:0}: Error finding container f9ba651468f7e7cbd3e6e79c733d6046f88f556f4c03accc9e71040e4d0889b7: Status 404 returned error can't find the container with id f9ba651468f7e7cbd3e6e79c733d6046f88f556f4c03accc9e71040e4d0889b7 Jul 24 15:14:53.142300 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:53.142266 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vrdjb"] Jul 24 15:14:53.145382 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:14:53.145356 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa859a6_6bbb_4318_a8fb_21b81df0c6e7.slice/crio-e0d2ae24d48d8a770ca528bb9fa1ca464bb2bcf14ed814f6e1777df104c3a2b3 WatchSource:0}: Error finding container e0d2ae24d48d8a770ca528bb9fa1ca464bb2bcf14ed814f6e1777df104c3a2b3: Status 404 returned error can't find the container with id e0d2ae24d48d8a770ca528bb9fa1ca464bb2bcf14ed814f6e1777df104c3a2b3 Jul 24 15:14:53.844301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:53.844263 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:54.062259 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.062180 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" event={"ID":"a0edaa9a-80ff-44f3-b066-d8285350c8fb","Type":"ContainerStarted","Data":"f9ba651468f7e7cbd3e6e79c733d6046f88f556f4c03accc9e71040e4d0889b7"} Jul 24 15:14:54.063625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.063592 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" event={"ID":"faa859a6-6bbb-4318-a8fb-21b81df0c6e7","Type":"ContainerStarted","Data":"e0d2ae24d48d8a770ca528bb9fa1ca464bb2bcf14ed814f6e1777df104c3a2b3"} Jul 24 15:14:54.066073 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.066011 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" event={"ID":"81080541-f392-46d6-a35d-0861c5f5aae2","Type":"ContainerStarted","Data":"255b1102e015def854f390f953385994c0665933fc3cbd499daa425b82103279"} Jul 24 15:14:54.068316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.068259 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" event={"ID":"96e8ac06-690b-4c08-8c7b-b9047c4f8b59","Type":"ContainerStarted","Data":"f7fe746292567bac6c7cd3f8a77556420bdc7db5f8c4c651b68f27815b50e04b"} Jul 24 15:14:54.068573 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.068518 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:14:54.095371 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.095256 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-587457bbc9-kwb8m" podStartSLOduration=4.087549132 podStartE2EDuration="7.09521532s" totalImagesPullingTime="3.007666188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:54.093877346 +0000 UTC m=+408.180529260" watchObservedRunningTime="2026-07-24 15:14:54.09521532 +0000 UTC m=+408.181867237" Jul 24 15:14:54.126645 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:54.126583 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" podStartSLOduration=2.683893169 podStartE2EDuration="6.126566258s" totalImagesPullingTime="3.442673089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:54.125314778 +0000 UTC m=+408.211966693" watchObservedRunningTime="2026-07-24 15:14:54.126566258 +0000 UTC m=+408.213218170" Jul 24 15:14:55.782842 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:55.782809 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:14:55.783132 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:55.782872 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:14:56.080398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.080300 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" event={"ID":"faa859a6-6bbb-4318-a8fb-21b81df0c6e7","Type":"ContainerStarted","Data":"d40657a8734262762d6179ab7d38a5cf669c6852f3574cb9362c23bd25e896df"} Jul 24 15:14:56.080398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.080373 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:14:56.081998 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.081972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" event={"ID":"a0edaa9a-80ff-44f3-b066-d8285350c8fb","Type":"ContainerStarted","Data":"7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410"} Jul 24 15:14:56.082191 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.082173 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:14:56.083618 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.083593 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zjrlv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:14:56.083716 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.083640 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:14:56.265148 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.265085 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" podStartSLOduration=3.635045272 podStartE2EDuration="6.265066102s" totalImagesPullingTime="2.63002083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:56.264305821 +0000 UTC m=+410.350957736" watchObservedRunningTime="2026-07-24 15:14:56.265066102 +0000 UTC m=+410.351718019" Jul 24 15:14:56.316278 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.316203 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" podStartSLOduration=2.6683312729999997 podStartE2EDuration="5.31618502s" totalImagesPullingTime="2.647853747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:56.313793091 +0000 UTC m=+410.400445007" watchObservedRunningTime="2026-07-24 15:14:56.31618502 +0000 UTC m=+410.402836934" Jul 24 15:14:56.676155 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.676115 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:56.784482 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:56.784440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:57.087224 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:57.087191 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zjrlv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:14:57.087412 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:57.087273 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:14:57.648473 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:57.648439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:58.020108 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:58.020073 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:14:58.338521 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:14:58.338430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:00.087607 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:00.087578 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:15:01.161502 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.161463 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:01.321327 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.321292 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj"] Jul 24 15:15:01.328016 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.327988 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.331161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.331130 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mt9hf\"" Jul 24 15:15:01.331808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.331788 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 15:15:01.348407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.348375 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj"] Jul 24 15:15:01.361650 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.361607 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp"] Jul 24 15:15:01.365551 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.365526 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:01.375032 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.375006 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kxwjh\"" Jul 24 15:15:01.375428 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.375401 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 15:15:01.390905 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.390870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4eea4eb5-4f44-4e8d-b36b-6967e1597d8b-cert\") pod \"kserve-controller-manager-d6f4b65b8-9pcbj\" (UID: \"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.391075 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.390916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfs7t\" (UniqueName: \"kubernetes.io/projected/4eea4eb5-4f44-4e8d-b36b-6967e1597d8b-kube-api-access-jfs7t\") pod \"kserve-controller-manager-d6f4b65b8-9pcbj\" (UID: \"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.396677 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.396646 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp"] Jul 24 15:15:01.479807 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.479700 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:01.491749 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.491712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779fddca-4ab7-4e51-b30d-244827b696fd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:01.491923 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.491765 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tbw\" (UniqueName: \"kubernetes.io/projected/779fddca-4ab7-4e51-b30d-244827b696fd-kube-api-access-26tbw\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:01.491923 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.491884 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4eea4eb5-4f44-4e8d-b36b-6967e1597d8b-cert\") pod \"kserve-controller-manager-d6f4b65b8-9pcbj\" (UID: \"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.492005 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.491943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfs7t\" (UniqueName: \"kubernetes.io/projected/4eea4eb5-4f44-4e8d-b36b-6967e1597d8b-kube-api-access-jfs7t\") pod \"kserve-controller-manager-d6f4b65b8-9pcbj\" (UID: \"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.494436 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.494417 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4eea4eb5-4f44-4e8d-b36b-6967e1597d8b-cert\") pod \"kserve-controller-manager-d6f4b65b8-9pcbj\" (UID: \"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.510806 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.510769 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfs7t\" (UniqueName: \"kubernetes.io/projected/4eea4eb5-4f44-4e8d-b36b-6967e1597d8b-kube-api-access-jfs7t\") pod \"kserve-controller-manager-d6f4b65b8-9pcbj\" (UID: \"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.592961 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.592915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779fddca-4ab7-4e51-b30d-244827b696fd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:01.592961 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.592968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tbw\" (UniqueName: \"kubernetes.io/projected/779fddca-4ab7-4e51-b30d-244827b696fd-kube-api-access-26tbw\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:01.593213 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:15:01.593068 2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 24 15:15:01.593213 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:15:01.593136 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/779fddca-4ab7-4e51-b30d-244827b696fd-cert podName:779fddca-4ab7-4e51-b30d-244827b696fd nodeName:}" failed. No retries permitted until 2026-07-24 15:15:02.093119124 +0000 UTC m=+416.179771015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/779fddca-4ab7-4e51-b30d-244827b696fd-cert") pod "llmisvc-controller-manager-777c6bd7d5-hztzp" (UID: "779fddca-4ab7-4e51-b30d-244827b696fd") : secret "llmisvc-webhook-server-cert" not found Jul 24 15:15:01.614062 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.614023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tbw\" (UniqueName: \"kubernetes.io/projected/779fddca-4ab7-4e51-b30d-244827b696fd-kube-api-access-26tbw\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:01.638389 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.638349 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:01.638587 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.638431 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:01.743540 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.743437 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:01.782808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.782774 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj"] Jul 24 15:15:01.788107 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:15:01.788080 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eea4eb5_4f44_4e8d_b36b_6967e1597d8b.slice/crio-a8f5e82dc1721296d61b2d0d3bab4fe9b0e926e994be2e502aacc2c944eac914 WatchSource:0}: Error finding container a8f5e82dc1721296d61b2d0d3bab4fe9b0e926e994be2e502aacc2c944eac914: Status 404 returned error can't find the container with id a8f5e82dc1721296d61b2d0d3bab4fe9b0e926e994be2e502aacc2c944eac914 Jul 24 15:15:01.927171 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:01.927116 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:02.096836 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:02.096794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779fddca-4ab7-4e51-b30d-244827b696fd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:02.099232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:02.099209 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/779fddca-4ab7-4e51-b30d-244827b696fd-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hztzp\" (UID: \"779fddca-4ab7-4e51-b30d-244827b696fd\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:02.111483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:02.111445 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" event={"ID":"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b","Type":"ContainerStarted","Data":"a8f5e82dc1721296d61b2d0d3bab4fe9b0e926e994be2e502aacc2c944eac914"} Jul 24 15:15:02.277255 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:02.277211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:02.445926 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:15:02.445888 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod779fddca_4ab7_4e51_b30d_244827b696fd.slice/crio-70b0be88205ca98e4ab9477c66063089715ed9f99c66735faed9a313f976110b WatchSource:0}: Error finding container 70b0be88205ca98e4ab9477c66063089715ed9f99c66735faed9a313f976110b: Status 404 returned error can't find the container with id 70b0be88205ca98e4ab9477c66063089715ed9f99c66735faed9a313f976110b Jul 24 15:15:02.447181 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:02.447156 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp"] Jul 24 15:15:03.116653 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:03.116611 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" event={"ID":"779fddca-4ab7-4e51-b30d-244827b696fd","Type":"ContainerStarted","Data":"70b0be88205ca98e4ab9477c66063089715ed9f99c66735faed9a313f976110b"} Jul 24 15:15:04.019455 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.019397 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv"] Jul 24 15:15:04.019918 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.019643 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerName="discovery" containerID="cri-o://7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410" gracePeriod=30 Jul 24 15:15:04.631443 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.631412 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:15:04.722821 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722744 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-kubeconfig\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.722821 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722820 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-dns-cert\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.723004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722849 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmtr9\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-kube-api-access-vmtr9\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.723004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722880 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0edaa9a-80ff-44f3-b066-d8285350c8fb-local-certs\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.723004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722917 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-token\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.723004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722959 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-ca-configmap\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.723004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.722995 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-cacerts\") pod \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\" (UID: \"a0edaa9a-80ff-44f3-b066-d8285350c8fb\") " Jul 24 15:15:04.724153 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.724116 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:15:04.726382 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.726355 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:15:04.726519 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.726416 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-kube-api-access-vmtr9" (OuterVolumeSpecName: "kube-api-access-vmtr9") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "kube-api-access-vmtr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:15:04.726519 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.726416 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:15:04.726519 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.726500 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-cacerts" (OuterVolumeSpecName: "cacerts") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:15:04.726703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.726597 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0edaa9a-80ff-44f3-b066-d8285350c8fb-local-certs" (OuterVolumeSpecName: "local-certs") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:15:04.727093 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.727060 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a0edaa9a-80ff-44f3-b066-d8285350c8fb" (UID: "a0edaa9a-80ff-44f3-b066-d8285350c8fb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:15:04.824061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824021 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-dns-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:04.824061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824056 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtr9\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-kube-api-access-vmtr9\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:04.824061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824066 2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0edaa9a-80ff-44f3-b066-d8285350c8fb-local-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:04.824316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824075 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:04.824316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824083 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-csr-ca-configmap\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:04.824316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824092 2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-cacerts\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:04.824316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:04.824100 2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0edaa9a-80ff-44f3-b066-d8285350c8fb-istio-kubeconfig\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:05.076753 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.076716 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-56ggm" Jul 24 15:15:05.129717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.129677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" event={"ID":"4eea4eb5-4f44-4e8d-b36b-6967e1597d8b","Type":"ContainerStarted","Data":"d485a90f3bd9e276970e634f49d1b2c3f09a6757b8c1922f133eb2844808ac9a"} Jul 24 15:15:05.129899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.129758 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:05.131386 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.131357 2563 generic.go:356] "Generic (PLEG): container finished" podID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerID="7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410" exitCode=0 Jul 24 15:15:05.131516 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.131402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" event={"ID":"a0edaa9a-80ff-44f3-b066-d8285350c8fb","Type":"ContainerDied","Data":"7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410"} Jul 24 15:15:05.131516 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.131437 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" event={"ID":"a0edaa9a-80ff-44f3-b066-d8285350c8fb","Type":"ContainerDied","Data":"f9ba651468f7e7cbd3e6e79c733d6046f88f556f4c03accc9e71040e4d0889b7"} Jul 24 15:15:05.131516 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.131457 2563 scope.go:117] "RemoveContainer" containerID="7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410" Jul 24 15:15:05.131516 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.131406 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv" Jul 24 15:15:05.188383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.187009 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" podStartSLOduration=1.370956284 podStartE2EDuration="4.186986897s" totalImagesPullingTime="2.816030613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:05.182959505 +0000 UTC m=+419.269611420" watchObservedRunningTime="2026-07-24 15:15:05.186986897 +0000 UTC m=+419.273638812" Jul 24 15:15:05.223849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.223807 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv"] Jul 24 15:15:05.235630 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.235596 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zjrlv"] Jul 24 15:15:05.567153 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.567123 2563 scope.go:117] "RemoveContainer" containerID="7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410" Jul 24 15:15:05.567534 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:15:05.567511 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410\": container with ID starting with 7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410 not found: ID does not exist" containerID="7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410" Jul 24 15:15:05.567617 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:05.567551 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410"} err="failed to get container status \"7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410\": rpc error: code = NotFound desc = could not find container \"7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410\": container with ID starting with 7c5420f16feef68055abf1de83434d8cff251eee4beba2fc44ffdb22a2574410 not found: ID does not exist" Jul 24 15:15:06.138704 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:06.138649 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" event={"ID":"779fddca-4ab7-4e51-b30d-244827b696fd","Type":"ContainerStarted","Data":"55e9853e678784f83615596d108d43f9799e681688a130d37c0ee5c6ff12e468"} Jul 24 15:15:06.139180 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:06.138967 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:06.166581 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:06.166524 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" podStartSLOduration=1.987104641 podStartE2EDuration="5.166505547s" totalImagesPullingTime="3.179400906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:06.164505691 +0000 UTC m=+420.251157605" watchObservedRunningTime="2026-07-24 15:15:06.166505547 +0000 UTC m=+420.253157459" Jul 24 15:15:06.438712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:06.438600 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" path="/var/lib/kubelet/pods/a0edaa9a-80ff-44f3-b066-d8285350c8fb/volumes" Jul 24 15:15:07.091107 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:07.091076 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vrdjb" Jul 24 15:15:36.144575 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:36.144511 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-9pcbj" Jul 24 15:15:37.146724 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:37.146691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hztzp" Jul 24 15:15:48.640115 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.640072 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:48.759431 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.759398 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:48.816783 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.816744 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fstjl"] Jul 24 15:15:48.817095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.817079 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerName="discovery" Jul 24 15:15:48.817095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.817096 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerName="discovery" Jul 24 15:15:48.817185 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.817162 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0edaa9a-80ff-44f3-b066-d8285350c8fb" containerName="discovery" Jul 24 15:15:48.827630 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.827594 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:48.830603 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.830567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-thqlr\"" Jul 24 15:15:48.831345 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.831309 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:15:48.831512 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.831309 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:15:48.833692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.833667 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fstjl"] Jul 24 15:15:48.888903 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.888851 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2np8d\" (UniqueName: \"kubernetes.io/projected/d90fd82e-47d4-458d-bb67-c85b31c10889-kube-api-access-2np8d\") pod \"kuadrant-operator-catalog-fstjl\" (UID: \"d90fd82e-47d4-458d-bb67-c85b31c10889\") " pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:48.900909 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.900822 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:15:48.990210 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:48.990171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2np8d\" (UniqueName: \"kubernetes.io/projected/d90fd82e-47d4-458d-bb67-c85b31c10889-kube-api-access-2np8d\") pod \"kuadrant-operator-catalog-fstjl\" (UID: \"d90fd82e-47d4-458d-bb67-c85b31c10889\") " pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:49.003266 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.003203 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2np8d\" (UniqueName: \"kubernetes.io/projected/d90fd82e-47d4-458d-bb67-c85b31c10889-kube-api-access-2np8d\") pod \"kuadrant-operator-catalog-fstjl\" (UID: \"d90fd82e-47d4-458d-bb67-c85b31c10889\") " pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:49.116959 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.116915 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fstjl"] Jul 24 15:15:49.117234 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.117217 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:49.271835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.271804 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fstjl"] Jul 24 15:15:49.274544 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:15:49.274504 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90fd82e_47d4_458d_bb67_c85b31c10889.slice/crio-a222f31ec7b7638f4426af5d2e6528b0f647410b5ae3662202ab3ae413db0946 WatchSource:0}: Error finding container a222f31ec7b7638f4426af5d2e6528b0f647410b5ae3662202ab3ae413db0946: Status 404 returned error can't find the container with id a222f31ec7b7638f4426af5d2e6528b0f647410b5ae3662202ab3ae413db0946 Jul 24 15:15:49.338845 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.338806 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" event={"ID":"d90fd82e-47d4-458d-bb67-c85b31c10889","Type":"ContainerStarted","Data":"a222f31ec7b7638f4426af5d2e6528b0f647410b5ae3662202ab3ae413db0946"} Jul 24 15:15:49.364591 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.364555 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jrbn9"] Jul 24 15:15:49.369101 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.369079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:49.380594 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.380567 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jrbn9"] Jul 24 15:15:49.495469 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.495362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4th4\" (UniqueName: \"kubernetes.io/projected/65b1c792-36c1-4105-9b52-3645e540167f-kube-api-access-t4th4\") pod \"kuadrant-operator-catalog-jrbn9\" (UID: \"65b1c792-36c1-4105-9b52-3645e540167f\") " pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:49.596057 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.596018 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4th4\" (UniqueName: \"kubernetes.io/projected/65b1c792-36c1-4105-9b52-3645e540167f-kube-api-access-t4th4\") pod \"kuadrant-operator-catalog-jrbn9\" (UID: \"65b1c792-36c1-4105-9b52-3645e540167f\") " pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:49.605598 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.605559 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4th4\" (UniqueName: \"kubernetes.io/projected/65b1c792-36c1-4105-9b52-3645e540167f-kube-api-access-t4th4\") pod \"kuadrant-operator-catalog-jrbn9\" (UID: \"65b1c792-36c1-4105-9b52-3645e540167f\") " pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:49.679692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.679647 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:49.831014 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:49.830984 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jrbn9"] Jul 24 15:15:49.881592 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:15:49.881557 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b1c792_36c1_4105_9b52_3645e540167f.slice/crio-fec47289694b77f58062cd1ee9dcbcc916d80393ba9b2ee7a833bf4f2a24e69b WatchSource:0}: Error finding container fec47289694b77f58062cd1ee9dcbcc916d80393ba9b2ee7a833bf4f2a24e69b: Status 404 returned error can't find the container with id fec47289694b77f58062cd1ee9dcbcc916d80393ba9b2ee7a833bf4f2a24e69b Jul 24 15:15:50.350749 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:50.350708 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" event={"ID":"65b1c792-36c1-4105-9b52-3645e540167f","Type":"ContainerStarted","Data":"fec47289694b77f58062cd1ee9dcbcc916d80393ba9b2ee7a833bf4f2a24e69b"} Jul 24 15:15:52.361473 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.361428 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" event={"ID":"d90fd82e-47d4-458d-bb67-c85b31c10889","Type":"ContainerStarted","Data":"c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d"} Jul 24 15:15:52.361982 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.361479 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" podUID="d90fd82e-47d4-458d-bb67-c85b31c10889" containerName="registry-server" containerID="cri-o://c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d" gracePeriod=2 Jul 24 15:15:52.363041 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.363015 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" event={"ID":"65b1c792-36c1-4105-9b52-3645e540167f","Type":"ContainerStarted","Data":"578624ccd02edd16e16a41b325bdffe5b899ce99d85883f1a982d9de6b21ef5a"} Jul 24 15:15:52.384620 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.384568 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" podStartSLOduration=2.280023512 podStartE2EDuration="4.384552985s" totalImagesPullingTime="2.104529473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:52.382041973 +0000 UTC m=+466.468693886" watchObservedRunningTime="2026-07-24 15:15:52.384552985 +0000 UTC m=+466.471204897" Jul 24 15:15:52.406163 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.406105 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" podStartSLOduration=1.9061581570000001 podStartE2EDuration="3.406086978s" totalImagesPullingTime="1.499928821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:52.40599902 +0000 UTC m=+466.492650932" watchObservedRunningTime="2026-07-24 15:15:52.406086978 +0000 UTC m=+466.492738890" Jul 24 15:15:52.608195 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.608168 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:52.724422 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.724328 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2np8d\" (UniqueName: \"kubernetes.io/projected/d90fd82e-47d4-458d-bb67-c85b31c10889-kube-api-access-2np8d\") pod \"d90fd82e-47d4-458d-bb67-c85b31c10889\" (UID: \"d90fd82e-47d4-458d-bb67-c85b31c10889\") " Jul 24 15:15:52.726574 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.726543 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90fd82e-47d4-458d-bb67-c85b31c10889-kube-api-access-2np8d" (OuterVolumeSpecName: "kube-api-access-2np8d") pod "d90fd82e-47d4-458d-bb67-c85b31c10889" (UID: "d90fd82e-47d4-458d-bb67-c85b31c10889"). InnerVolumeSpecName "kube-api-access-2np8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:15:52.825288 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:52.825221 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2np8d\" (UniqueName: \"kubernetes.io/projected/d90fd82e-47d4-458d-bb67-c85b31c10889-kube-api-access-2np8d\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:15:53.370553 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.370512 2563 generic.go:356] "Generic (PLEG): container finished" podID="d90fd82e-47d4-458d-bb67-c85b31c10889" containerID="c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d" exitCode=0 Jul 24 15:15:53.371055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.370559 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" Jul 24 15:15:53.371055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.370623 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" event={"ID":"d90fd82e-47d4-458d-bb67-c85b31c10889","Type":"ContainerDied","Data":"c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d"} Jul 24 15:15:53.371055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.370678 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fstjl" event={"ID":"d90fd82e-47d4-458d-bb67-c85b31c10889","Type":"ContainerDied","Data":"a222f31ec7b7638f4426af5d2e6528b0f647410b5ae3662202ab3ae413db0946"} Jul 24 15:15:53.371055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.370698 2563 scope.go:117] "RemoveContainer" containerID="c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d" Jul 24 15:15:53.380298 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.380277 2563 scope.go:117] "RemoveContainer" containerID="c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d" Jul 24 15:15:53.380604 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:15:53.380584 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d\": container with ID starting with c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d not found: ID does not exist" containerID="c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d" Jul 24 15:15:53.380653 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.380613 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d"} err="failed to get container status \"c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d\": rpc error: code = NotFound desc = could not find container \"c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d\": container with ID starting with c0a7abd7b3dda5a4bfeb70b509991a66cf2f10d57bb642378f57c8145de51a2d not found: ID does not exist" Jul 24 15:15:53.396147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.396113 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fstjl"] Jul 24 15:15:53.399235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:53.399205 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fstjl"] Jul 24 15:15:54.436025 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:54.435987 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90fd82e-47d4-458d-bb67-c85b31c10889" path="/var/lib/kubelet/pods/d90fd82e-47d4-458d-bb67-c85b31c10889/volumes" Jul 24 15:15:59.680325 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:59.680278 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:59.680766 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:59.680352 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:15:59.702114 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:15:59.702084 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:16:00.430087 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:00.430060 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jrbn9" Jul 24 15:16:04.387063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.387019 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh"] Jul 24 15:16:04.387489 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.387418 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90fd82e-47d4-458d-bb67-c85b31c10889" containerName="registry-server" Jul 24 15:16:04.387489 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.387434 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90fd82e-47d4-458d-bb67-c85b31c10889" containerName="registry-server" Jul 24 15:16:04.387489 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.387490 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90fd82e-47d4-458d-bb67-c85b31c10889" containerName="registry-server" Jul 24 15:16:04.390470 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.390450 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.393102 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.393079 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h4mzg\"" Jul 24 15:16:04.401717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.401689 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh"] Jul 24 15:16:04.524017 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.523972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.524017 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.524024 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ll2v\" (UniqueName: \"kubernetes.io/projected/ba270f6a-06cb-48f9-830d-96595a1e515a-kube-api-access-5ll2v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.524236 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.524106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.625527 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.625487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.625691 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.625533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ll2v\" (UniqueName: \"kubernetes.io/projected/ba270f6a-06cb-48f9-830d-96595a1e515a-kube-api-access-5ll2v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.625691 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.625620 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.625965 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.625942 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.625965 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.625958 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.634646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.634609 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ll2v\" (UniqueName: \"kubernetes.io/projected/ba270f6a-06cb-48f9-830d-96595a1e515a-kube-api-access-5ll2v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.700103 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.699995 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:04.796936 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.796897 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv"] Jul 24 15:16:04.802046 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.802021 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:04.809867 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.809832 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv"] Jul 24 15:16:04.843337 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.843310 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh"] Jul 24 15:16:04.845405 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:04.845379 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba270f6a_06cb_48f9_830d_96595a1e515a.slice/crio-0350ab9a49b52831b14e69b56aa6b4fe5c9141d720d969b4f8de2e483c3b127f WatchSource:0}: Error finding container 0350ab9a49b52831b14e69b56aa6b4fe5c9141d720d969b4f8de2e483c3b127f: Status 404 returned error can't find the container with id 0350ab9a49b52831b14e69b56aa6b4fe5c9141d720d969b4f8de2e483c3b127f Jul 24 15:16:04.928177 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.928141 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:04.928177 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.928195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzzc\" (UniqueName: \"kubernetes.io/projected/474a2157-dd26-47d3-b13c-8f44f8cbbd81-kube-api-access-wqzzc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:04.928416 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:04.928384 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.029627 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.029562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzzc\" (UniqueName: \"kubernetes.io/projected/474a2157-dd26-47d3-b13c-8f44f8cbbd81-kube-api-access-wqzzc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.029826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.029671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.029826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.029697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.030022 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.030007 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.030075 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.030056 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.039124 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.039095 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzzc\" (UniqueName: \"kubernetes.io/projected/474a2157-dd26-47d3-b13c-8f44f8cbbd81-kube-api-access-wqzzc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.115378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.115334 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:05.193612 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.193577 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs"] Jul 24 15:16:05.198215 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.198187 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.210534 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.210483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs"] Jul 24 15:16:05.260957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.260922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv"] Jul 24 15:16:05.263623 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:05.263595 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a2157_dd26_47d3_b13c_8f44f8cbbd81.slice/crio-fcfa33d1cf4b2c693a8ebbf32197b130b71881ddac6a35b1f1f77ecf510f0d93 WatchSource:0}: Error finding container fcfa33d1cf4b2c693a8ebbf32197b130b71881ddac6a35b1f1f77ecf510f0d93: Status 404 returned error can't find the container with id fcfa33d1cf4b2c693a8ebbf32197b130b71881ddac6a35b1f1f77ecf510f0d93 Jul 24 15:16:05.332458 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.332427 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.332458 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.332466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtqs\" (UniqueName: \"kubernetes.io/projected/2e2500b0-53b4-40d0-ba33-1b4315246a10-kube-api-access-svtqs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.332689 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.332580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.430320 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.430289 2563 generic.go:356] "Generic (PLEG): container finished" podID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerID="8f5c24862fb2e5cfe750fa74839728972f04c50ce0067700c3ca095218a502c3" exitCode=0 Jul 24 15:16:05.430778 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.430317 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" event={"ID":"474a2157-dd26-47d3-b13c-8f44f8cbbd81","Type":"ContainerDied","Data":"8f5c24862fb2e5cfe750fa74839728972f04c50ce0067700c3ca095218a502c3"} Jul 24 15:16:05.430778 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.430363 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" event={"ID":"474a2157-dd26-47d3-b13c-8f44f8cbbd81","Type":"ContainerStarted","Data":"fcfa33d1cf4b2c693a8ebbf32197b130b71881ddac6a35b1f1f77ecf510f0d93"} Jul 24 15:16:05.432109 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.432087 2563 generic.go:356] "Generic (PLEG): container finished" podID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerID="6ba3de27e16199488d7c5dff89f149c698428cdfa0fd88948cd22281d6d9e53a" exitCode=0 Jul 24 15:16:05.432199 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.432121 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" event={"ID":"ba270f6a-06cb-48f9-830d-96595a1e515a","Type":"ContainerDied","Data":"6ba3de27e16199488d7c5dff89f149c698428cdfa0fd88948cd22281d6d9e53a"} Jul 24 15:16:05.432199 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.432157 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" event={"ID":"ba270f6a-06cb-48f9-830d-96595a1e515a","Type":"ContainerStarted","Data":"0350ab9a49b52831b14e69b56aa6b4fe5c9141d720d969b4f8de2e483c3b127f"} Jul 24 15:16:05.433020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.433001 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.433122 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.433033 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtqs\" (UniqueName: \"kubernetes.io/projected/2e2500b0-53b4-40d0-ba33-1b4315246a10-kube-api-access-svtqs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.433122 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.433081 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.433410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.433389 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.433450 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.433426 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.441301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.441277 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtqs\" (UniqueName: \"kubernetes.io/projected/2e2500b0-53b4-40d0-ba33-1b4315246a10-kube-api-access-svtqs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.511655 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.511609 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:05.587811 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.587772 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9"] Jul 24 15:16:05.592584 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.592552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.602830 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.602783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9"] Jul 24 15:16:05.650536 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.650508 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs"] Jul 24 15:16:05.655481 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:05.653307 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2500b0_53b4_40d0_ba33_1b4315246a10.slice/crio-af3ad83c28b216c5d030ced8d617f7e9542cb6a7f988c4dada0c5d78d2cc35e4 WatchSource:0}: Error finding container af3ad83c28b216c5d030ced8d617f7e9542cb6a7f988c4dada0c5d78d2cc35e4: Status 404 returned error can't find the container with id af3ad83c28b216c5d030ced8d617f7e9542cb6a7f988c4dada0c5d78d2cc35e4 Jul 24 15:16:05.736546 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.736498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tn8d\" (UniqueName: \"kubernetes.io/projected/6acd7206-7243-4478-a9f6-b403b597a44a-kube-api-access-7tn8d\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.736724 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.736566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.736724 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.736639 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.837761 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.837666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tn8d\" (UniqueName: \"kubernetes.io/projected/6acd7206-7243-4478-a9f6-b403b597a44a-kube-api-access-7tn8d\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.837761 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.837725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.837761 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.837750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.838083 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.838067 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.838138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.838103 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.847836 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.847808 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tn8d\" (UniqueName: \"kubernetes.io/projected/6acd7206-7243-4478-a9f6-b403b597a44a-kube-api-access-7tn8d\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:05.904945 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:05.904905 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:06.042567 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.042532 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9"] Jul 24 15:16:06.045943 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:06.045912 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acd7206_7243_4478_a9f6_b403b597a44a.slice/crio-429c2eb9110af39c276f3d4a8bc6d5381b6ec9cd74d10198cd5bb6b6d7be9e19 WatchSource:0}: Error finding container 429c2eb9110af39c276f3d4a8bc6d5381b6ec9cd74d10198cd5bb6b6d7be9e19: Status 404 returned error can't find the container with id 429c2eb9110af39c276f3d4a8bc6d5381b6ec9cd74d10198cd5bb6b6d7be9e19 Jul 24 15:16:06.442353 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.442314 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerID="059067149b274bcab93b7f8c3b90a92bcbdcd8adfe7aa91ec4b0c4898e9d86fd" exitCode=0 Jul 24 15:16:06.442842 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.442391 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" event={"ID":"2e2500b0-53b4-40d0-ba33-1b4315246a10","Type":"ContainerDied","Data":"059067149b274bcab93b7f8c3b90a92bcbdcd8adfe7aa91ec4b0c4898e9d86fd"} Jul 24 15:16:06.442842 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.442452 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" event={"ID":"2e2500b0-53b4-40d0-ba33-1b4315246a10","Type":"ContainerStarted","Data":"af3ad83c28b216c5d030ced8d617f7e9542cb6a7f988c4dada0c5d78d2cc35e4"} Jul 24 15:16:06.443996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.443974 2563 generic.go:356] "Generic (PLEG): container finished" podID="6acd7206-7243-4478-a9f6-b403b597a44a" containerID="9aa729648a9694adeb514147615e2d6fdc1e726452d1760a660c23664cfb3c76" exitCode=0 Jul 24 15:16:06.444097 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.444030 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" event={"ID":"6acd7206-7243-4478-a9f6-b403b597a44a","Type":"ContainerDied","Data":"9aa729648a9694adeb514147615e2d6fdc1e726452d1760a660c23664cfb3c76"} Jul 24 15:16:06.444097 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:06.444067 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" event={"ID":"6acd7206-7243-4478-a9f6-b403b597a44a","Type":"ContainerStarted","Data":"429c2eb9110af39c276f3d4a8bc6d5381b6ec9cd74d10198cd5bb6b6d7be9e19"} Jul 24 15:16:07.451407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:07.451368 2563 generic.go:356] "Generic (PLEG): container finished" podID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerID="c57b2169b07db284a555721d3978db0d7da1e224749e44bea4f000d0562371e0" exitCode=0 Jul 24 15:16:07.451916 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:07.451448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" event={"ID":"474a2157-dd26-47d3-b13c-8f44f8cbbd81","Type":"ContainerDied","Data":"c57b2169b07db284a555721d3978db0d7da1e224749e44bea4f000d0562371e0"} Jul 24 15:16:07.453591 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:07.453567 2563 generic.go:356] "Generic (PLEG): container finished" podID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerID="05588d205e03616af202a4f6f32b6b13e57273a8f437f38e8de92a5b84a0c551" exitCode=0 Jul 24 15:16:07.453696 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:07.453643 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" event={"ID":"ba270f6a-06cb-48f9-830d-96595a1e515a","Type":"ContainerDied","Data":"05588d205e03616af202a4f6f32b6b13e57273a8f437f38e8de92a5b84a0c551"} Jul 24 15:16:08.461203 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.461163 2563 generic.go:356] "Generic (PLEG): container finished" podID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerID="bfa3c49402fe594296937b24216a32ad3faab42c999a8682a3968d37db3f9532" exitCode=0 Jul 24 15:16:08.461652 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.461259 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" event={"ID":"474a2157-dd26-47d3-b13c-8f44f8cbbd81","Type":"ContainerDied","Data":"bfa3c49402fe594296937b24216a32ad3faab42c999a8682a3968d37db3f9532"} Jul 24 15:16:08.463077 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.463052 2563 generic.go:356] "Generic (PLEG): container finished" podID="6acd7206-7243-4478-a9f6-b403b597a44a" containerID="3d12d93d0f768e2f8673371cd54068f6c0da279d736f651243151aff51e9a0db" exitCode=0 Jul 24 15:16:08.463225 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.463140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" event={"ID":"6acd7206-7243-4478-a9f6-b403b597a44a","Type":"ContainerDied","Data":"3d12d93d0f768e2f8673371cd54068f6c0da279d736f651243151aff51e9a0db"} Jul 24 15:16:08.465467 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.465446 2563 generic.go:356] "Generic (PLEG): container finished" podID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerID="5aa45d5ec4fb740a467205b005295cfb6036b86708e68add678e83c4b6b4eec6" exitCode=0 Jul 24 15:16:08.465557 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.465480 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" event={"ID":"ba270f6a-06cb-48f9-830d-96595a1e515a","Type":"ContainerDied","Data":"5aa45d5ec4fb740a467205b005295cfb6036b86708e68add678e83c4b6b4eec6"} Jul 24 15:16:08.467405 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.467380 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerID="eb01d77d2dbc777d34b2a340badbd0e6f71125f85384557492041904aeb554cb" exitCode=0 Jul 24 15:16:08.467483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:08.467404 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" event={"ID":"2e2500b0-53b4-40d0-ba33-1b4315246a10","Type":"ContainerDied","Data":"eb01d77d2dbc777d34b2a340badbd0e6f71125f85384557492041904aeb554cb"} Jul 24 15:16:09.474699 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.474658 2563 generic.go:356] "Generic (PLEG): container finished" podID="6acd7206-7243-4478-a9f6-b403b597a44a" containerID="33b2301de8116e7c7c37ddde90d1fd698dcb1cde3eabcd00ce85bb67a7e7c485" exitCode=0 Jul 24 15:16:09.475124 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.474744 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" event={"ID":"6acd7206-7243-4478-a9f6-b403b597a44a","Type":"ContainerDied","Data":"33b2301de8116e7c7c37ddde90d1fd698dcb1cde3eabcd00ce85bb67a7e7c485"} Jul 24 15:16:09.476859 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.476835 2563 generic.go:356] "Generic (PLEG): container finished" podID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerID="a52063d69a2a201923417e4bc2507aacb2daab34a608f0e3660d32bd1508cb64" exitCode=0 Jul 24 15:16:09.476992 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.476913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" event={"ID":"2e2500b0-53b4-40d0-ba33-1b4315246a10","Type":"ContainerDied","Data":"a52063d69a2a201923417e4bc2507aacb2daab34a608f0e3660d32bd1508cb64"} Jul 24 15:16:09.609063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.609035 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:09.649208 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.649168 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:09.771833 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.771803 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-util\") pod \"ba270f6a-06cb-48f9-830d-96595a1e515a\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " Jul 24 15:16:09.772001 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.771887 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-util\") pod \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " Jul 24 15:16:09.772001 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.771921 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-bundle\") pod \"ba270f6a-06cb-48f9-830d-96595a1e515a\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " Jul 24 15:16:09.772001 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.771979 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ll2v\" (UniqueName: \"kubernetes.io/projected/ba270f6a-06cb-48f9-830d-96595a1e515a-kube-api-access-5ll2v\") pod \"ba270f6a-06cb-48f9-830d-96595a1e515a\" (UID: \"ba270f6a-06cb-48f9-830d-96595a1e515a\") " Jul 24 15:16:09.772164 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.772002 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqzzc\" (UniqueName: \"kubernetes.io/projected/474a2157-dd26-47d3-b13c-8f44f8cbbd81-kube-api-access-wqzzc\") pod \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " Jul 24 15:16:09.772164 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.772026 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-bundle\") pod \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\" (UID: \"474a2157-dd26-47d3-b13c-8f44f8cbbd81\") " Jul 24 15:16:09.772620 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.772585 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-bundle" (OuterVolumeSpecName: "bundle") pod "ba270f6a-06cb-48f9-830d-96595a1e515a" (UID: "ba270f6a-06cb-48f9-830d-96595a1e515a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:09.772761 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.772656 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-bundle" (OuterVolumeSpecName: "bundle") pod "474a2157-dd26-47d3-b13c-8f44f8cbbd81" (UID: "474a2157-dd26-47d3-b13c-8f44f8cbbd81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:09.774147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.774115 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474a2157-dd26-47d3-b13c-8f44f8cbbd81-kube-api-access-wqzzc" (OuterVolumeSpecName: "kube-api-access-wqzzc") pod "474a2157-dd26-47d3-b13c-8f44f8cbbd81" (UID: "474a2157-dd26-47d3-b13c-8f44f8cbbd81"). InnerVolumeSpecName "kube-api-access-wqzzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:16:09.774277 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.774220 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba270f6a-06cb-48f9-830d-96595a1e515a-kube-api-access-5ll2v" (OuterVolumeSpecName: "kube-api-access-5ll2v") pod "ba270f6a-06cb-48f9-830d-96595a1e515a" (UID: "ba270f6a-06cb-48f9-830d-96595a1e515a"). InnerVolumeSpecName "kube-api-access-5ll2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:16:09.777517 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.777482 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-util" (OuterVolumeSpecName: "util") pod "ba270f6a-06cb-48f9-830d-96595a1e515a" (UID: "ba270f6a-06cb-48f9-830d-96595a1e515a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:09.777627 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.777608 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-util" (OuterVolumeSpecName: "util") pod "474a2157-dd26-47d3-b13c-8f44f8cbbd81" (UID: "474a2157-dd26-47d3-b13c-8f44f8cbbd81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:09.872949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.872905 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ll2v\" (UniqueName: \"kubernetes.io/projected/ba270f6a-06cb-48f9-830d-96595a1e515a-kube-api-access-5ll2v\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:09.872949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.872940 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzzc\" (UniqueName: \"kubernetes.io/projected/474a2157-dd26-47d3-b13c-8f44f8cbbd81-kube-api-access-wqzzc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:09.872949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.872950 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:09.872949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.872958 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:09.873225 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.872967 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/474a2157-dd26-47d3-b13c-8f44f8cbbd81-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:09.873225 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:09.872975 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba270f6a-06cb-48f9-830d-96595a1e515a-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:10.483695 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.483663 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" Jul 24 15:16:10.484113 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.483662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv" event={"ID":"474a2157-dd26-47d3-b13c-8f44f8cbbd81","Type":"ContainerDied","Data":"fcfa33d1cf4b2c693a8ebbf32197b130b71881ddac6a35b1f1f77ecf510f0d93"} Jul 24 15:16:10.484113 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.483821 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcfa33d1cf4b2c693a8ebbf32197b130b71881ddac6a35b1f1f77ecf510f0d93" Jul 24 15:16:10.485715 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.485690 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" Jul 24 15:16:10.485715 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.485697 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh" event={"ID":"ba270f6a-06cb-48f9-830d-96595a1e515a","Type":"ContainerDied","Data":"0350ab9a49b52831b14e69b56aa6b4fe5c9141d720d969b4f8de2e483c3b127f"} Jul 24 15:16:10.485935 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.485765 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0350ab9a49b52831b14e69b56aa6b4fe5c9141d720d969b4f8de2e483c3b127f" Jul 24 15:16:10.642460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.642428 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:10.644534 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.644515 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:10.783665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.783629 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tn8d\" (UniqueName: \"kubernetes.io/projected/6acd7206-7243-4478-a9f6-b403b597a44a-kube-api-access-7tn8d\") pod \"6acd7206-7243-4478-a9f6-b403b597a44a\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " Jul 24 15:16:10.783858 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.783691 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-util\") pod \"6acd7206-7243-4478-a9f6-b403b597a44a\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " Jul 24 15:16:10.783858 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.783732 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-bundle\") pod \"2e2500b0-53b4-40d0-ba33-1b4315246a10\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " Jul 24 15:16:10.784009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.783907 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-bundle\") pod \"6acd7206-7243-4478-a9f6-b403b597a44a\" (UID: \"6acd7206-7243-4478-a9f6-b403b597a44a\") " Jul 24 15:16:10.784009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.783962 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svtqs\" (UniqueName: \"kubernetes.io/projected/2e2500b0-53b4-40d0-ba33-1b4315246a10-kube-api-access-svtqs\") pod \"2e2500b0-53b4-40d0-ba33-1b4315246a10\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " Jul 24 15:16:10.784009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.783992 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-util\") pod \"2e2500b0-53b4-40d0-ba33-1b4315246a10\" (UID: \"2e2500b0-53b4-40d0-ba33-1b4315246a10\") " Jul 24 15:16:10.784383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.784355 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-bundle" (OuterVolumeSpecName: "bundle") pod "2e2500b0-53b4-40d0-ba33-1b4315246a10" (UID: "2e2500b0-53b4-40d0-ba33-1b4315246a10"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:10.784728 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.784704 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-bundle" (OuterVolumeSpecName: "bundle") pod "6acd7206-7243-4478-a9f6-b403b597a44a" (UID: "6acd7206-7243-4478-a9f6-b403b597a44a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:10.785857 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.785836 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6acd7206-7243-4478-a9f6-b403b597a44a-kube-api-access-7tn8d" (OuterVolumeSpecName: "kube-api-access-7tn8d") pod "6acd7206-7243-4478-a9f6-b403b597a44a" (UID: "6acd7206-7243-4478-a9f6-b403b597a44a"). InnerVolumeSpecName "kube-api-access-7tn8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:16:10.786333 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.786313 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2500b0-53b4-40d0-ba33-1b4315246a10-kube-api-access-svtqs" (OuterVolumeSpecName: "kube-api-access-svtqs") pod "2e2500b0-53b4-40d0-ba33-1b4315246a10" (UID: "2e2500b0-53b4-40d0-ba33-1b4315246a10"). InnerVolumeSpecName "kube-api-access-svtqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:16:10.790095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.790048 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-util" (OuterVolumeSpecName: "util") pod "2e2500b0-53b4-40d0-ba33-1b4315246a10" (UID: "2e2500b0-53b4-40d0-ba33-1b4315246a10"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:10.790751 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.790461 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-util" (OuterVolumeSpecName: "util") pod "6acd7206-7243-4478-a9f6-b403b597a44a" (UID: "6acd7206-7243-4478-a9f6-b403b597a44a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:16:10.885632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.885585 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tn8d\" (UniqueName: \"kubernetes.io/projected/6acd7206-7243-4478-a9f6-b403b597a44a-kube-api-access-7tn8d\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:10.885632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.885622 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:10.885632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.885635 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:10.885632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.885645 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6acd7206-7243-4478-a9f6-b403b597a44a-bundle\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:10.885896 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.885656 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtqs\" (UniqueName: \"kubernetes.io/projected/2e2500b0-53b4-40d0-ba33-1b4315246a10-kube-api-access-svtqs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:10.885896 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:10.885667 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e2500b0-53b4-40d0-ba33-1b4315246a10-util\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:16:11.492205 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:11.492177 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" Jul 24 15:16:11.492641 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:11.492218 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9" event={"ID":"6acd7206-7243-4478-a9f6-b403b597a44a","Type":"ContainerDied","Data":"429c2eb9110af39c276f3d4a8bc6d5381b6ec9cd74d10198cd5bb6b6d7be9e19"} Jul 24 15:16:11.492641 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:11.492279 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="429c2eb9110af39c276f3d4a8bc6d5381b6ec9cd74d10198cd5bb6b6d7be9e19" Jul 24 15:16:11.494193 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:11.494169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" event={"ID":"2e2500b0-53b4-40d0-ba33-1b4315246a10","Type":"ContainerDied","Data":"af3ad83c28b216c5d030ced8d617f7e9542cb6a7f988c4dada0c5d78d2cc35e4"} Jul 24 15:16:11.494327 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:11.494210 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3ad83c28b216c5d030ced8d617f7e9542cb6a7f988c4dada0c5d78d2cc35e4" Jul 24 15:16:11.494327 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:11.494176 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs" Jul 24 15:16:17.343112 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:17.343064 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:19.947877 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.947837 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w"] Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948149 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948161 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948169 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948176 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948187 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="pull" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948193 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="pull" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948200 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="pull" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948205 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="pull" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948212 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="util" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948217 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="util" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948224 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948230 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948253 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="pull" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948259 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="pull" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948269 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948274 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="extract" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948282 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="util" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948287 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="util" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948293 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="util" Jul 24 15:16:19.948287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948298 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="util" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948304 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="util" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948309 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="util" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948314 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="pull" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948319 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="pull" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948388 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6acd7206-7243-4478-a9f6-b403b597a44a" containerName="extract" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948397 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2500b0-53b4-40d0-ba33-1b4315246a10" containerName="extract" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948403 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="474a2157-dd26-47d3-b13c-8f44f8cbbd81" containerName="extract" Jul 24 15:16:19.948835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.948411 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba270f6a-06cb-48f9-830d-96595a1e515a" containerName="extract" Jul 24 15:16:19.952524 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.952504 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:19.956020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.955997 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bqcjr\"" Jul 24 15:16:19.960690 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.960668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rkp\" (UniqueName: \"kubernetes.io/projected/4774f989-ae25-4fe8-a904-7e931251bd97-kube-api-access-m8rkp\") pod \"limitador-operator-controller-manager-859d7b78bd-6np5w\" (UID: \"4774f989-ae25-4fe8-a904-7e931251bd97\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:19.977867 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:19.977830 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w"] Jul 24 15:16:20.061205 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:20.061164 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rkp\" (UniqueName: \"kubernetes.io/projected/4774f989-ae25-4fe8-a904-7e931251bd97-kube-api-access-m8rkp\") pod \"limitador-operator-controller-manager-859d7b78bd-6np5w\" (UID: \"4774f989-ae25-4fe8-a904-7e931251bd97\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:20.078986 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:20.078952 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rkp\" (UniqueName: \"kubernetes.io/projected/4774f989-ae25-4fe8-a904-7e931251bd97-kube-api-access-m8rkp\") pod \"limitador-operator-controller-manager-859d7b78bd-6np5w\" (UID: \"4774f989-ae25-4fe8-a904-7e931251bd97\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:20.262044 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:20.261997 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:20.434161 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:20.434130 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4774f989_ae25_4fe8_a904_7e931251bd97.slice/crio-eb691182f150305c52f171185114d5176f09ed3c5afa8c7dd0a35e3f2a49580e WatchSource:0}: Error finding container eb691182f150305c52f171185114d5176f09ed3c5afa8c7dd0a35e3f2a49580e: Status 404 returned error can't find the container with id eb691182f150305c52f171185114d5176f09ed3c5afa8c7dd0a35e3f2a49580e Jul 24 15:16:20.437033 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:20.437012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w"] Jul 24 15:16:20.543979 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:20.543887 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" event={"ID":"4774f989-ae25-4fe8-a904-7e931251bd97","Type":"ContainerStarted","Data":"eb691182f150305c52f171185114d5176f09ed3c5afa8c7dd0a35e3f2a49580e"} Jul 24 15:16:20.565694 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:20.565660 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:21.328120 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:21.328080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:22.558020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:22.557976 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" event={"ID":"4774f989-ae25-4fe8-a904-7e931251bd97","Type":"ContainerStarted","Data":"3b9e14549df010e5a8fc6716ebc1a726f0bc838191a4b60bf51d5c6cf203ec78"} Jul 24 15:16:22.558725 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:22.558057 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:22.594892 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:22.594840 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" podStartSLOduration=2.046105654 podStartE2EDuration="3.594822497s" totalImagesPullingTime="1.548716843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:16:22.590563159 +0000 UTC m=+496.677215071" watchObservedRunningTime="2026-07-24 15:16:22.594822497 +0000 UTC m=+496.681474411" Jul 24 15:16:24.133378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.133341 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8"] Jul 24 15:16:24.136866 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.136844 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:24.139640 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.139609 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jrpvc\"" Jul 24 15:16:24.151362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.151335 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8"] Jul 24 15:16:24.195745 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.195703 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wpcd\" (UniqueName: \"kubernetes.io/projected/e368c3f7-4432-421c-af39-0f02cf74a0c7-kube-api-access-8wpcd\") pod \"authorino-operator-75cc8d5cb7-fr4h8\" (UID: \"e368c3f7-4432-421c-af39-0f02cf74a0c7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:24.296425 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.296386 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wpcd\" (UniqueName: \"kubernetes.io/projected/e368c3f7-4432-421c-af39-0f02cf74a0c7-kube-api-access-8wpcd\") pod \"authorino-operator-75cc8d5cb7-fr4h8\" (UID: \"e368c3f7-4432-421c-af39-0f02cf74a0c7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:24.315595 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.315564 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wpcd\" (UniqueName: \"kubernetes.io/projected/e368c3f7-4432-421c-af39-0f02cf74a0c7-kube-api-access-8wpcd\") pod \"authorino-operator-75cc8d5cb7-fr4h8\" (UID: \"e368c3f7-4432-421c-af39-0f02cf74a0c7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:24.447726 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.447642 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:24.600077 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.600051 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8"] Jul 24 15:16:24.601590 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:24.601560 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode368c3f7_4432_421c_af39_0f02cf74a0c7.slice/crio-eb7e3c1a24735864e1bec1c3d4c2cb5dc6ef6d97658b28450c74f62a8f45380d WatchSource:0}: Error finding container eb7e3c1a24735864e1bec1c3d4c2cb5dc6ef6d97658b28450c74f62a8f45380d: Status 404 returned error can't find the container with id eb7e3c1a24735864e1bec1c3d4c2cb5dc6ef6d97658b28450c74f62a8f45380d Jul 24 15:16:24.808014 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:24.807971 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:25.197595 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:25.197495 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:25.576746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:25.576706 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" event={"ID":"e368c3f7-4432-421c-af39-0f02cf74a0c7","Type":"ContainerStarted","Data":"eb7e3c1a24735864e1bec1c3d4c2cb5dc6ef6d97658b28450c74f62a8f45380d"} Jul 24 15:16:26.562745 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:26.562698 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:27.099323 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.099296 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65"] Jul 24 15:16:27.102699 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.102678 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:27.109385 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.109358 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 15:16:27.111280 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.111254 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h2d9z\"" Jul 24 15:16:27.118829 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.118803 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65"] Jul 24 15:16:27.121358 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.121335 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8d2k\" (UniqueName: \"kubernetes.io/projected/394da736-b485-443b-9d74-e72df7a5f17e-kube-api-access-x8d2k\") pod \"dns-operator-controller-manager-864dd78c4c-nsd65\" (UID: \"394da736-b485-443b-9d74-e72df7a5f17e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:27.222695 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.222650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8d2k\" (UniqueName: \"kubernetes.io/projected/394da736-b485-443b-9d74-e72df7a5f17e-kube-api-access-x8d2k\") pod \"dns-operator-controller-manager-864dd78c4c-nsd65\" (UID: \"394da736-b485-443b-9d74-e72df7a5f17e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:27.237812 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.237781 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8d2k\" (UniqueName: \"kubernetes.io/projected/394da736-b485-443b-9d74-e72df7a5f17e-kube-api-access-x8d2k\") pod \"dns-operator-controller-manager-864dd78c4c-nsd65\" (UID: \"394da736-b485-443b-9d74-e72df7a5f17e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:27.414500 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.414406 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:27.558615 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.558518 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65"] Jul 24 15:16:27.561155 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:27.561127 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394da736_b485_443b_9d74_e72df7a5f17e.slice/crio-29d12715649b18d812cd49bae6c91c21d1aeb019820abe2da39ee3d4d927559e WatchSource:0}: Error finding container 29d12715649b18d812cd49bae6c91c21d1aeb019820abe2da39ee3d4d927559e: Status 404 returned error can't find the container with id 29d12715649b18d812cd49bae6c91c21d1aeb019820abe2da39ee3d4d927559e Jul 24 15:16:27.591455 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.591417 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" event={"ID":"394da736-b485-443b-9d74-e72df7a5f17e","Type":"ContainerStarted","Data":"29d12715649b18d812cd49bae6c91c21d1aeb019820abe2da39ee3d4d927559e"} Jul 24 15:16:27.593088 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.593031 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" event={"ID":"e368c3f7-4432-421c-af39-0f02cf74a0c7","Type":"ContainerStarted","Data":"697fab9778dc456694b1805cf8d259c72e1956d70877380f4d169154611012f2"} Jul 24 15:16:27.593217 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.593126 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:27.618399 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.618322 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" podStartSLOduration=1.192005832 podStartE2EDuration="3.618306513s" totalImagesPullingTime="2.426300681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:16:27.61744962 +0000 UTC m=+501.704101532" watchObservedRunningTime="2026-07-24 15:16:27.618306513 +0000 UTC m=+501.704958425" Jul 24 15:16:27.940723 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:27.940689 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:28.066292 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:28.066238 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:29.102421 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:29.102382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:29.223819 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:29.223788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:29.856091 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:29.856059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:30.615448 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:30.615402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" event={"ID":"394da736-b485-443b-9d74-e72df7a5f17e","Type":"ContainerStarted","Data":"283ebbc4d00c27c752c6b5d208d6937a1d096c84e1e97c3742c1628708935db1"} Jul 24 15:16:30.615889 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:30.615477 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:30.643104 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:30.643052 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" podStartSLOduration=1.590821351 podStartE2EDuration="3.643035647s" totalImagesPullingTime="2.052214296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:16:30.641043505 +0000 UTC m=+504.727695419" watchObservedRunningTime="2026-07-24 15:16:30.643035647 +0000 UTC m=+504.729687561" Jul 24 15:16:31.929670 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:31.929634 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj"] Jul 24 15:16:31.932690 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:31.932667 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:31.936947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:31.936926 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56vlm\"" Jul 24 15:16:31.949357 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:31.949326 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj"] Jul 24 15:16:31.964731 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:31.964694 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pldw\" (UniqueName: \"kubernetes.io/projected/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-kube-api-access-8pldw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:31.964908 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:31.964773 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:32.065697 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.065652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pldw\" (UniqueName: \"kubernetes.io/projected/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-kube-api-access-8pldw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:32.065857 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.065721 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:32.066079 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.066062 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:32.074694 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.074666 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pldw\" (UniqueName: \"kubernetes.io/projected/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-kube-api-access-8pldw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:32.242721 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.242630 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:32.388736 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.388708 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj"] Jul 24 15:16:32.390419 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:32.390381 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f35ae3_9a84_41a7_a04d_46e0ffa1db3e.slice/crio-8382304018afc91dccb1226b66582136c86b517479d4f92f300559b67f528d3a WatchSource:0}: Error finding container 8382304018afc91dccb1226b66582136c86b517479d4f92f300559b67f528d3a: Status 404 returned error can't find the container with id 8382304018afc91dccb1226b66582136c86b517479d4f92f300559b67f528d3a Jul 24 15:16:32.627235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.627195 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" event={"ID":"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e","Type":"ContainerStarted","Data":"8382304018afc91dccb1226b66582136c86b517479d4f92f300559b67f528d3a"} Jul 24 15:16:32.923509 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:32.923423 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:33.025914 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:33.025874 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:33.566119 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:33.566084 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:38.353594 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.353513 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:38.479899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.479854 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:38.605713 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.605625 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-fr4h8" Jul 24 15:16:38.663912 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.663869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" event={"ID":"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e","Type":"ContainerStarted","Data":"498e853c1bbf1e9f1a9aec9b903d6e2e30a82c4a3f363fb42c94361040def4e6"} Jul 24 15:16:38.664113 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.664023 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:38.688093 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.688039 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" podStartSLOduration=1.623859989 podStartE2EDuration="7.688020651s" totalImagesPullingTime="6.064160662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:16:38.686081501 +0000 UTC m=+512.772733415" watchObservedRunningTime="2026-07-24 15:16:38.688020651 +0000 UTC m=+512.774672564" Jul 24 15:16:38.710710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.710662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:38.814304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:38.814264 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:41.329926 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.329891 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv"] Jul 24 15:16:41.334952 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.334924 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.337356 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.337324 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 15:16:41.337507 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.337368 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 15:16:41.337507 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.337433 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h4mzg\"" Jul 24 15:16:41.342511 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.342476 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv"] Jul 24 15:16:41.356747 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.356712 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:41.450046 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.450008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49eb213-b09d-40fd-871a-03f5796356b7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.450231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.450108 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b49eb213-b09d-40fd-871a-03f5796356b7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.450231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.450128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdtv\" (UniqueName: \"kubernetes.io/projected/b49eb213-b09d-40fd-871a-03f5796356b7-kube-api-access-bqdtv\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.551232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.551184 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b49eb213-b09d-40fd-871a-03f5796356b7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.551232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.551238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdtv\" (UniqueName: \"kubernetes.io/projected/b49eb213-b09d-40fd-871a-03f5796356b7-kube-api-access-bqdtv\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.551536 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.551334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49eb213-b09d-40fd-871a-03f5796356b7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.551602 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:41.551537 2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 15:16:41.551655 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:41.551602 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b49eb213-b09d-40fd-871a-03f5796356b7-plugin-serving-cert podName:b49eb213-b09d-40fd-871a-03f5796356b7 nodeName:}" failed. No retries permitted until 2026-07-24 15:16:42.05158255 +0000 UTC m=+516.138234446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b49eb213-b09d-40fd-871a-03f5796356b7-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-f6jsv" (UID: "b49eb213-b09d-40fd-871a-03f5796356b7") : secret "plugin-serving-cert" not found Jul 24 15:16:41.551966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.551947 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b49eb213-b09d-40fd-871a-03f5796356b7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.560463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.560432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdtv\" (UniqueName: \"kubernetes.io/projected/b49eb213-b09d-40fd-871a-03f5796356b7-kube-api-access-bqdtv\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:41.623806 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.623727 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nsd65" Jul 24 15:16:41.829032 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:41.828991 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:42.056181 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.056126 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49eb213-b09d-40fd-871a-03f5796356b7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:42.058808 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.058779 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b49eb213-b09d-40fd-871a-03f5796356b7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-f6jsv\" (UID: \"b49eb213-b09d-40fd-871a-03f5796356b7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:42.244913 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.244871 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" Jul 24 15:16:42.392051 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.392024 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv"] Jul 24 15:16:42.393331 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:16:42.393291 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb49eb213_b09d_40fd_871a_03f5796356b7.slice/crio-d7c437670a16b3a001edab71e4fd457df9e0d1c5e5b8bc8f26592f33f0509deb WatchSource:0}: Error finding container d7c437670a16b3a001edab71e4fd457df9e0d1c5e5b8bc8f26592f33f0509deb: Status 404 returned error can't find the container with id d7c437670a16b3a001edab71e4fd457df9e0d1c5e5b8bc8f26592f33f0509deb Jul 24 15:16:42.684307 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.684160 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:42.688023 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.687993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" event={"ID":"b49eb213-b09d-40fd-871a-03f5796356b7","Type":"ContainerStarted","Data":"d7c437670a16b3a001edab71e4fd457df9e0d1c5e5b8bc8f26592f33f0509deb"} Jul 24 15:16:42.817139 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:42.817099 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:46.073863 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:46.073815 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:46.175007 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:46.174962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:49.672458 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:49.672404 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:51.657012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.656974 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj"] Jul 24 15:16:51.657556 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.657209 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" containerName="manager" containerID="cri-o://498e853c1bbf1e9f1a9aec9b903d6e2e30a82c4a3f363fb42c94361040def4e6" gracePeriod=2 Jul 24 15:16:51.675511 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:51.675458 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:51.676537 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.676497 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj"] Jul 24 15:16:51.695886 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.695849 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w"] Jul 24 15:16:51.696142 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.696114 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" containerName="manager" containerID="cri-o://3b9e14549df010e5a8fc6716ebc1a726f0bc838191a4b60bf51d5c6cf203ec78" gracePeriod=2 Jul 24 15:16:51.769766 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.769317 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl"] Jul 24 15:16:51.769953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.769789 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" containerName="manager" Jul 24 15:16:51.769953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.769810 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" containerName="manager" Jul 24 15:16:51.769953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.769927 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" containerName="manager" Jul 24 15:16:51.771186 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:51.771137 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:51.773597 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.773572 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w"] Jul 24 15:16:51.773722 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.773663 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:51.805028 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.804996 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl"] Jul 24 15:16:51.809543 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.809506 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf"] Jul 24 15:16:51.809951 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.809933 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" containerName="manager" Jul 24 15:16:51.809951 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.809952 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" containerName="manager" Jul 24 15:16:51.810112 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.810045 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" containerName="manager" Jul 24 15:16:51.814569 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.814536 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:16:51.838508 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.838469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf"] Jul 24 15:16:51.855111 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.855071 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r8skl\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:51.855324 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.855186 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxqh\" (UniqueName: \"kubernetes.io/projected/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-kube-api-access-tsxqh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r8skl\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:51.865839 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:51.865720 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6np5w\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:51.877880 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:51.877838 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:51.880532 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:51.880489 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6np5w\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:16:51.884286 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:16:51.884233 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:16:51.956347 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.956234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ndr\" (UniqueName: \"kubernetes.io/projected/533263ef-432c-41af-a089-7db19363fc2b-kube-api-access-r2ndr\") pod \"limitador-operator-controller-manager-859d7b78bd-sklnf\" (UID: \"533263ef-432c-41af-a089-7db19363fc2b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:16:51.956347 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.956321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r8skl\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:51.956594 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.956371 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxqh\" (UniqueName: \"kubernetes.io/projected/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-kube-api-access-tsxqh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r8skl\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:51.956712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.956687 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r8skl\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:51.975440 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:51.975400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxqh\" (UniqueName: \"kubernetes.io/projected/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-kube-api-access-tsxqh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-r8skl\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:52.057350 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:52.057311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ndr\" (UniqueName: \"kubernetes.io/projected/533263ef-432c-41af-a089-7db19363fc2b-kube-api-access-r2ndr\") pod \"limitador-operator-controller-manager-859d7b78bd-sklnf\" (UID: \"533263ef-432c-41af-a089-7db19363fc2b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:16:52.094898 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:52.094863 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ndr\" (UniqueName: \"kubernetes.io/projected/533263ef-432c-41af-a089-7db19363fc2b-kube-api-access-r2ndr\") pod \"limitador-operator-controller-manager-859d7b78bd-sklnf\" (UID: \"533263ef-432c-41af-a089-7db19363fc2b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:16:52.103658 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:52.103611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:16:52.128059 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:52.128019 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:16:53.482877 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:53.482834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:16:53.595962 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:16:53.595918 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:03.830707 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.830623 2563 generic.go:356] "Generic (PLEG): container finished" podID="4774f989-ae25-4fe8-a904-7e931251bd97" containerID="3b9e14549df010e5a8fc6716ebc1a726f0bc838191a4b60bf51d5c6cf203ec78" exitCode=0 Jul 24 15:17:03.842509 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.842471 2563 generic.go:356] "Generic (PLEG): container finished" podID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" containerID="498e853c1bbf1e9f1a9aec9b903d6e2e30a82c4a3f363fb42c94361040def4e6" exitCode=0 Jul 24 15:17:03.942998 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.942968 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:17:03.948279 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:03.948222 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:17:03.952541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.952508 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:17:03.956909 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:17:03.956879 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533263ef_432c_41af_a089_7db19363fc2b.slice/crio-f0b0fc47addb0c80cf97be6bcaa361890ce16dc7702b8081ea511fd7265ad5e9 WatchSource:0}: Error finding container f0b0fc47addb0c80cf97be6bcaa361890ce16dc7702b8081ea511fd7265ad5e9: Status 404 returned error can't find the container with id f0b0fc47addb0c80cf97be6bcaa361890ce16dc7702b8081ea511fd7265ad5e9 Jul 24 15:17:03.961456 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:03.959051 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6np5w\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:17:03.961456 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.960773 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf"] Jul 24 15:17:03.965735 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:03.965699 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:17:03.970014 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.969986 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pldw\" (UniqueName: \"kubernetes.io/projected/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-kube-api-access-8pldw\") pod \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " Jul 24 15:17:03.970094 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.970050 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-extensions-socket-volume\") pod \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\" (UID: \"e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e\") " Jul 24 15:17:03.970130 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.970102 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8rkp\" (UniqueName: \"kubernetes.io/projected/4774f989-ae25-4fe8-a904-7e931251bd97-kube-api-access-m8rkp\") pod \"4774f989-ae25-4fe8-a904-7e931251bd97\" (UID: \"4774f989-ae25-4fe8-a904-7e931251bd97\") " Jul 24 15:17:03.970584 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.970552 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" (UID: "e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:17:03.972365 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.972335 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-kube-api-access-8pldw" (OuterVolumeSpecName: "kube-api-access-8pldw") pod "e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" (UID: "e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e"). InnerVolumeSpecName "kube-api-access-8pldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:17:03.972463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:03.972393 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4774f989-ae25-4fe8-a904-7e931251bd97-kube-api-access-m8rkp" (OuterVolumeSpecName: "kube-api-access-m8rkp") pod "4774f989-ae25-4fe8-a904-7e931251bd97" (UID: "4774f989-ae25-4fe8-a904-7e931251bd97"). InnerVolumeSpecName "kube-api-access-m8rkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:17:04.001023 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.000997 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl"] Jul 24 15:17:04.004157 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:17:04.004131 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5cf736_3252_4eed_8e95_58b9a9b2c31a.slice/crio-bba07c8e21caba9f76d09909262792ab31df7ca4b33258fc8b58e9413b1d3abf WatchSource:0}: Error finding container bba07c8e21caba9f76d09909262792ab31df7ca4b33258fc8b58e9413b1d3abf: Status 404 returned error can't find the container with id bba07c8e21caba9f76d09909262792ab31df7ca4b33258fc8b58e9413b1d3abf Jul 24 15:17:04.070948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.070914 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-extensions-socket-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:04.070948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.070943 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rkp\" (UniqueName: \"kubernetes.io/projected/4774f989-ae25-4fe8-a904-7e931251bd97-kube-api-access-m8rkp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:04.070948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.070953 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pldw\" (UniqueName: \"kubernetes.io/projected/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e-kube-api-access-8pldw\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:04.436922 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.436873 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" path="/var/lib/kubelet/pods/4774f989-ae25-4fe8-a904-7e931251bd97/volumes" Jul 24 15:17:04.437357 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.437336 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" path="/var/lib/kubelet/pods/e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e/volumes" Jul 24 15:17:04.850130 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.850089 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" event={"ID":"533263ef-432c-41af-a089-7db19363fc2b","Type":"ContainerStarted","Data":"235d05cedbb64cce46838e286c97bf1d874de3253936c0d05993fdd887f34df6"} Jul 24 15:17:04.850646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.850146 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" event={"ID":"533263ef-432c-41af-a089-7db19363fc2b","Type":"ContainerStarted","Data":"f0b0fc47addb0c80cf97be6bcaa361890ce16dc7702b8081ea511fd7265ad5e9"} Jul 24 15:17:04.850646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.850217 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:17:04.851649 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.851630 2563 scope.go:117] "RemoveContainer" containerID="3b9e14549df010e5a8fc6716ebc1a726f0bc838191a4b60bf51d5c6cf203ec78" Jul 24 15:17:04.851649 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.851643 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:17:04.853170 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.853135 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:17:04.855193 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.854910 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" event={"ID":"b49eb213-b09d-40fd-871a-03f5796356b7","Type":"ContainerStarted","Data":"f1b2a3b113ea5e054ec29acfb61bf8cd840da7ce6ed051473331eaf709ef1be6"} Jul 24 15:17:04.857052 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.857031 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" event={"ID":"ae5cf736-3252-4eed-8e95-58b9a9b2c31a","Type":"ContainerStarted","Data":"fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2"} Jul 24 15:17:04.857152 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.857071 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" event={"ID":"ae5cf736-3252-4eed-8e95-58b9a9b2c31a","Type":"ContainerStarted","Data":"bba07c8e21caba9f76d09909262792ab31df7ca4b33258fc8b58e9413b1d3abf"} Jul 24 15:17:04.857152 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.857092 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:17:04.861436 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.861414 2563 scope.go:117] "RemoveContainer" containerID="498e853c1bbf1e9f1a9aec9b903d6e2e30a82c4a3f363fb42c94361040def4e6" Jul 24 15:17:04.889311 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:04.889257 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:17:04.890121 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.890077 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" podStartSLOduration=13.890061612 podStartE2EDuration="13.890061612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:04.887037584 +0000 UTC m=+538.973689498" watchObservedRunningTime="2026-07-24 15:17:04.890061612 +0000 UTC m=+538.976713551" Jul 24 15:17:04.910152 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.910093 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-f6jsv" podStartSLOduration=2.442005752 podStartE2EDuration="23.91007609s" totalImagesPullingTime="21.468070338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:04.90903309 +0000 UTC m=+538.995685004" watchObservedRunningTime="2026-07-24 15:17:04.91007609 +0000 UTC m=+538.996728003" Jul 24 15:17:04.963940 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:04.963863 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" podStartSLOduration=13.963827536 podStartE2EDuration="13.963827536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:04.958279044 +0000 UTC m=+539.044930958" watchObservedRunningTime="2026-07-24 15:17:04.963827536 +0000 UTC m=+539.050479451" Jul 24 15:17:04.964217 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:04.964190 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6np5w\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:17:06.437526 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:06.437490 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7s4dj\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e1f35ae3-9a84-41a7-a04d-46e0ffa1db3e" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7s4dj" Jul 24 15:17:06.439636 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:06.439608 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-6np5w\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="4774f989-ae25-4fe8-a904-7e931251bd97" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6np5w" Jul 24 15:17:08.437384 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.437350 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl"] Jul 24 15:17:08.437816 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.437530 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" podUID="ae5cf736-3252-4eed-8e95-58b9a9b2c31a" containerName="manager" containerID="cri-o://fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2" gracePeriod=10 Jul 24 15:17:08.701208 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.701097 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:17:08.813896 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.813861 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsxqh\" (UniqueName: \"kubernetes.io/projected/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-kube-api-access-tsxqh\") pod \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " Jul 24 15:17:08.814102 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.813910 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-extensions-socket-volume\") pod \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\" (UID: \"ae5cf736-3252-4eed-8e95-58b9a9b2c31a\") " Jul 24 15:17:08.814330 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.814303 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ae5cf736-3252-4eed-8e95-58b9a9b2c31a" (UID: "ae5cf736-3252-4eed-8e95-58b9a9b2c31a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:17:08.816036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.816008 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-kube-api-access-tsxqh" (OuterVolumeSpecName: "kube-api-access-tsxqh") pod "ae5cf736-3252-4eed-8e95-58b9a9b2c31a" (UID: "ae5cf736-3252-4eed-8e95-58b9a9b2c31a"). InnerVolumeSpecName "kube-api-access-tsxqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:17:08.881429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.881393 2563 generic.go:356] "Generic (PLEG): container finished" podID="ae5cf736-3252-4eed-8e95-58b9a9b2c31a" containerID="fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2" exitCode=0 Jul 24 15:17:08.881429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.881441 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" Jul 24 15:17:08.881692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.881482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" event={"ID":"ae5cf736-3252-4eed-8e95-58b9a9b2c31a","Type":"ContainerDied","Data":"fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2"} Jul 24 15:17:08.881692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.881537 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl" event={"ID":"ae5cf736-3252-4eed-8e95-58b9a9b2c31a","Type":"ContainerDied","Data":"bba07c8e21caba9f76d09909262792ab31df7ca4b33258fc8b58e9413b1d3abf"} Jul 24 15:17:08.881692 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.881562 2563 scope.go:117] "RemoveContainer" containerID="fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2" Jul 24 15:17:08.891053 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.891028 2563 scope.go:117] "RemoveContainer" containerID="fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2" Jul 24 15:17:08.891474 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:17:08.891432 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2\": container with ID starting with fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2 not found: ID does not exist" containerID="fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2" Jul 24 15:17:08.891562 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.891480 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2"} err="failed to get container status \"fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2\": rpc error: code = NotFound desc = could not find container \"fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2\": container with ID starting with fcb7e7fa71081999f8aa8e0202621423ecfd1f5997c8496513045f131939ffd2 not found: ID does not exist" Jul 24 15:17:08.905596 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.905564 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl"] Jul 24 15:17:08.914733 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.914698 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-r8skl"] Jul 24 15:17:08.914900 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.914876 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxqh\" (UniqueName: \"kubernetes.io/projected/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-kube-api-access-tsxqh\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:08.914900 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:08.914898 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae5cf736-3252-4eed-8e95-58b9a9b2c31a-extensions-socket-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:10.436996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:10.436959 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5cf736-3252-4eed-8e95-58b9a9b2c31a" path="/var/lib/kubelet/pods/ae5cf736-3252-4eed-8e95-58b9a9b2c31a/volumes" Jul 24 15:17:15.017482 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:15.017439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:15.102145 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:15.102108 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:15.867386 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:15.867356 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sklnf" Jul 24 15:17:24.891599 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:24.891557 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f"] Jul 24 15:17:24.892081 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:24.891923 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5cf736-3252-4eed-8e95-58b9a9b2c31a" containerName="manager" Jul 24 15:17:24.892081 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:24.891934 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5cf736-3252-4eed-8e95-58b9a9b2c31a" containerName="manager" Jul 24 15:17:24.892081 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:24.891991 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5cf736-3252-4eed-8e95-58b9a9b2c31a" containerName="manager" Jul 24 15:17:24.914693 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:24.914601 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:25.029584 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.029533 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f"] Jul 24 15:17:25.029584 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.029563 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.044370 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.044343 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 15:17:25.044557 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.044396 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4gtql\"" Jul 24 15:17:25.161063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.160952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.160997 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161086 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5060b3-a279-4958-af38-7bcff776112b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5060b3-a279-4958-af38-7bcff776112b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161318 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dkl\" (UniqueName: \"kubernetes.io/projected/bf5060b3-a279-4958-af38-7bcff776112b-kube-api-access-q8dkl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161533 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5060b3-a279-4958-af38-7bcff776112b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.161533 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.161388 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262749 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262763 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262789 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5060b3-a279-4958-af38-7bcff776112b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5060b3-a279-4958-af38-7bcff776112b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.262976 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.262973 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dkl\" (UniqueName: \"kubernetes.io/projected/bf5060b3-a279-4958-af38-7bcff776112b-kube-api-access-q8dkl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.263460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.263008 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5060b3-a279-4958-af38-7bcff776112b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.263460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.263217 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.263460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.263350 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.263460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.263453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.263680 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.263502 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.263868 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.263840 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf5060b3-a279-4958-af38-7bcff776112b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.265179 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.265160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf5060b3-a279-4958-af38-7bcff776112b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.265328 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.265313 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf5060b3-a279-4958-af38-7bcff776112b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.273464 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.273433 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf5060b3-a279-4958-af38-7bcff776112b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.273595 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.273536 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dkl\" (UniqueName: \"kubernetes.io/projected/bf5060b3-a279-4958-af38-7bcff776112b-kube-api-access-q8dkl\") pod \"maas-default-gateway-openshift-default-78bb78ff78-v8z9f\" (UID: \"bf5060b3-a279-4958-af38-7bcff776112b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.277300 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.277274 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:25.340611 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.340567 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:25.531873 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.531836 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f"] Jul 24 15:17:25.534569 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:17:25.534543 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5060b3_a279_4958_af38_7bcff776112b.slice/crio-354fb4ee31f92c0b9d58a2827c53c1c52df59bdeb98c893cc82b07b74bb46492 WatchSource:0}: Error finding container 354fb4ee31f92c0b9d58a2827c53c1c52df59bdeb98c893cc82b07b74bb46492: Status 404 returned error can't find the container with id 354fb4ee31f92c0b9d58a2827c53c1c52df59bdeb98c893cc82b07b74bb46492 Jul 24 15:17:25.975451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:25.975410 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" event={"ID":"bf5060b3-a279-4958-af38-7bcff776112b","Type":"ContainerStarted","Data":"354fb4ee31f92c0b9d58a2827c53c1c52df59bdeb98c893cc82b07b74bb46492"} Jul 24 15:17:28.813919 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:28.813878 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:17:28.814293 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:28.813954 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:17:28.814293 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:28.814004 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:17:29.001054 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:29.001015 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" event={"ID":"bf5060b3-a279-4958-af38-7bcff776112b","Type":"ContainerStarted","Data":"9d90d6578377d338eaa41fb53ef38f8427f9ba90009917c0a78aee4a578bbbf5"} Jul 24 15:17:29.031120 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:29.031047 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" podStartSLOduration=1.7542136849999999 podStartE2EDuration="5.031022327s" totalImagesPullingTime="3.276808642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:29.028423291 +0000 UTC m=+563.115075205" watchObservedRunningTime="2026-07-24 15:17:29.031022327 +0000 UTC m=+563.117674246" Jul 24 15:17:29.047038 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:29.047003 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:29.341301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:29.341235 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:29.346883 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:29.346849 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:30.007258 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.007207 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:30.008139 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.008119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:17:30.185138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.185097 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:30.187902 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.187852 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:30.248809 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.248776 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:30.248978 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.248878 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.252299 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.252271 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 15:17:30.299452 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.299417 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:30.310389 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.310345 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:30.406076 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.406040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmms\" (UniqueName: \"kubernetes.io/projected/17b12064-c945-468e-9db2-e37e5ef2221b-kube-api-access-nwmms\") pod \"limitador-limitador-64c94fd64f-vv627\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.406076 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.406079 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17b12064-c945-468e-9db2-e37e5ef2221b-config-file\") pod \"limitador-limitador-64c94fd64f-vv627\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.432069 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.432036 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:30.512447 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.507402 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmms\" (UniqueName: \"kubernetes.io/projected/17b12064-c945-468e-9db2-e37e5ef2221b-kube-api-access-nwmms\") pod \"limitador-limitador-64c94fd64f-vv627\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.512447 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.507479 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17b12064-c945-468e-9db2-e37e5ef2221b-config-file\") pod \"limitador-limitador-64c94fd64f-vv627\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.512447 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.508412 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17b12064-c945-468e-9db2-e37e5ef2221b-config-file\") pod \"limitador-limitador-64c94fd64f-vv627\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.516928 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.516896 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmms\" (UniqueName: \"kubernetes.io/projected/17b12064-c945-468e-9db2-e37e5ef2221b-kube-api-access-nwmms\") pod \"limitador-limitador-64c94fd64f-vv627\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.562178 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.562144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:30.752125 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:30.752096 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:30.755163 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:17:30.755122 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b12064_c945_468e_9db2_e37e5ef2221b.slice/crio-e521fc2d18c44ea377e6b1f2cc839d1018cd5de6033407ee7c661b1d8af1055c WatchSource:0}: Error finding container e521fc2d18c44ea377e6b1f2cc839d1018cd5de6033407ee7c661b1d8af1055c: Status 404 returned error can't find the container with id e521fc2d18c44ea377e6b1f2cc839d1018cd5de6033407ee7c661b1d8af1055c Jul 24 15:17:31.014334 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.014284 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" event={"ID":"17b12064-c945-468e-9db2-e37e5ef2221b","Type":"ContainerStarted","Data":"e521fc2d18c44ea377e6b1f2cc839d1018cd5de6033407ee7c661b1d8af1055c"} Jul 24 15:17:31.142913 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.142875 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:31.245350 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.245317 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:31.279511 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.279425 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-tlmgk"] Jul 24 15:17:31.421858 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.421820 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tlmgk"] Jul 24 15:17:31.422012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.421921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:17:31.424791 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.424760 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vlnbj\"" Jul 24 15:17:31.515267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.515214 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqxfj\" (UniqueName: \"kubernetes.io/projected/75560229-c689-4fbe-a0b6-2cbdaeab83a5-kube-api-access-qqxfj\") pod \"authorino-75d556bd77-tlmgk\" (UID: \"75560229-c689-4fbe-a0b6-2cbdaeab83a5\") " pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:17:31.616215 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.616132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqxfj\" (UniqueName: \"kubernetes.io/projected/75560229-c689-4fbe-a0b6-2cbdaeab83a5-kube-api-access-qqxfj\") pod \"authorino-75d556bd77-tlmgk\" (UID: \"75560229-c689-4fbe-a0b6-2cbdaeab83a5\") " pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:17:31.631457 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.631426 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqxfj\" (UniqueName: \"kubernetes.io/projected/75560229-c689-4fbe-a0b6-2cbdaeab83a5-kube-api-access-qqxfj\") pod \"authorino-75d556bd77-tlmgk\" (UID: \"75560229-c689-4fbe-a0b6-2cbdaeab83a5\") " pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:17:31.734042 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.734006 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:17:31.927584 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:31.924875 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tlmgk"] Jul 24 15:17:31.927584 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:17:31.927410 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75560229_c689_4fbe_a0b6_2cbdaeab83a5.slice/crio-56be7d9c5b700e13c660aa11f65314ddbdad7082575a4b7d0f04bb533ccb7563 WatchSource:0}: Error finding container 56be7d9c5b700e13c660aa11f65314ddbdad7082575a4b7d0f04bb533ccb7563: Status 404 returned error can't find the container with id 56be7d9c5b700e13c660aa11f65314ddbdad7082575a4b7d0f04bb533ccb7563 Jul 24 15:17:32.023804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:32.023760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tlmgk" event={"ID":"75560229-c689-4fbe-a0b6-2cbdaeab83a5","Type":"ContainerStarted","Data":"56be7d9c5b700e13c660aa11f65314ddbdad7082575a4b7d0f04bb533ccb7563"} Jul 24 15:17:36.051507 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:36.051468 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" event={"ID":"17b12064-c945-468e-9db2-e37e5ef2221b","Type":"ContainerStarted","Data":"87d5be8caaa2037a478a4d4ac619c5fa84cb3fd3e44fe3c6d2f6c61d177cc06f"} Jul 24 15:17:36.051977 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:36.051592 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:36.053160 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:36.053134 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tlmgk" event={"ID":"75560229-c689-4fbe-a0b6-2cbdaeab83a5","Type":"ContainerStarted","Data":"95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0"} Jul 24 15:17:36.112753 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:36.112692 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" podStartSLOduration=1.1839591 podStartE2EDuration="6.112674754s" totalImagesPullingTime="4.928715654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:36.10962433 +0000 UTC m=+570.196276354" watchObservedRunningTime="2026-07-24 15:17:36.112674754 +0000 UTC m=+570.199326667" Jul 24 15:17:36.135560 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:36.135506 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-tlmgk" podStartSLOduration=1.377582677 podStartE2EDuration="5.135489547s" totalImagesPullingTime="3.75790687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:36.132787783 +0000 UTC m=+570.219439695" watchObservedRunningTime="2026-07-24 15:17:36.135489547 +0000 UTC m=+570.222141460" Jul 24 15:17:47.059561 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:47.059525 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:47.207670 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:47.207629 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:47.207861 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:47.207811 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" podUID="17b12064-c945-468e-9db2-e37e5ef2221b" containerName="limitador" containerID="cri-o://87d5be8caaa2037a478a4d4ac619c5fa84cb3fd3e44fe3c6d2f6c61d177cc06f" gracePeriod=30 Jul 24 15:17:48.122604 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.122569 2563 generic.go:356] "Generic (PLEG): container finished" podID="17b12064-c945-468e-9db2-e37e5ef2221b" containerID="87d5be8caaa2037a478a4d4ac619c5fa84cb3fd3e44fe3c6d2f6c61d177cc06f" exitCode=0 Jul 24 15:17:48.123009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.122654 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" event={"ID":"17b12064-c945-468e-9db2-e37e5ef2221b","Type":"ContainerDied","Data":"87d5be8caaa2037a478a4d4ac619c5fa84cb3fd3e44fe3c6d2f6c61d177cc06f"} Jul 24 15:17:48.160407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.160382 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:48.180897 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.180866 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwmms\" (UniqueName: \"kubernetes.io/projected/17b12064-c945-468e-9db2-e37e5ef2221b-kube-api-access-nwmms\") pod \"17b12064-c945-468e-9db2-e37e5ef2221b\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " Jul 24 15:17:48.181050 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.180935 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17b12064-c945-468e-9db2-e37e5ef2221b-config-file\") pod \"17b12064-c945-468e-9db2-e37e5ef2221b\" (UID: \"17b12064-c945-468e-9db2-e37e5ef2221b\") " Jul 24 15:17:48.181329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.181290 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b12064-c945-468e-9db2-e37e5ef2221b-config-file" (OuterVolumeSpecName: "config-file") pod "17b12064-c945-468e-9db2-e37e5ef2221b" (UID: "17b12064-c945-468e-9db2-e37e5ef2221b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:17:48.183024 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.183002 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b12064-c945-468e-9db2-e37e5ef2221b-kube-api-access-nwmms" (OuterVolumeSpecName: "kube-api-access-nwmms") pod "17b12064-c945-468e-9db2-e37e5ef2221b" (UID: "17b12064-c945-468e-9db2-e37e5ef2221b"). InnerVolumeSpecName "kube-api-access-nwmms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:17:48.281624 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.281535 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmms\" (UniqueName: \"kubernetes.io/projected/17b12064-c945-468e-9db2-e37e5ef2221b-kube-api-access-nwmms\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:48.281624 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:48.281564 2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/17b12064-c945-468e-9db2-e37e5ef2221b-config-file\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:17:49.129917 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:49.129885 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" Jul 24 15:17:49.129917 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:49.129900 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vv627" event={"ID":"17b12064-c945-468e-9db2-e37e5ef2221b","Type":"ContainerDied","Data":"e521fc2d18c44ea377e6b1f2cc839d1018cd5de6033407ee7c661b1d8af1055c"} Jul 24 15:17:49.130444 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:49.129959 2563 scope.go:117] "RemoveContainer" containerID="87d5be8caaa2037a478a4d4ac619c5fa84cb3fd3e44fe3c6d2f6c61d177cc06f" Jul 24 15:17:49.153714 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:49.153663 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:49.158035 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:49.157998 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vv627"] Jul 24 15:17:50.437666 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:50.437633 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b12064-c945-468e-9db2-e37e5ef2221b" path="/var/lib/kubelet/pods/17b12064-c945-468e-9db2-e37e5ef2221b/volumes" Jul 24 15:17:51.389617 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.389587 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:51.495971 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.495937 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:51.824583 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.824550 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pn29f"] Jul 24 15:17:51.824904 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.824892 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b12064-c945-468e-9db2-e37e5ef2221b" containerName="limitador" Jul 24 15:17:51.824955 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.824906 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b12064-c945-468e-9db2-e37e5ef2221b" containerName="limitador" Jul 24 15:17:51.824994 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.824963 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b12064-c945-468e-9db2-e37e5ef2221b" containerName="limitador" Jul 24 15:17:51.827570 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.827552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:51.831291 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.831231 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 15:17:51.831450 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.831352 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:17:51.831879 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.831833 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 15:17:51.832056 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.832039 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rmggt\"" Jul 24 15:17:51.840320 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.840292 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pn29f"] Jul 24 15:17:51.857219 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.857182 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:17:51.910146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.910100 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/09cd3231-8871-42cc-a1b2-f0826c378703-data\") pod \"postgres-69fb4594cf-pn29f\" (UID: \"09cd3231-8871-42cc-a1b2-f0826c378703\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:51.910348 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:51.910172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6gm\" (UniqueName: \"kubernetes.io/projected/09cd3231-8871-42cc-a1b2-f0826c378703-kube-api-access-vc6gm\") pod \"postgres-69fb4594cf-pn29f\" (UID: \"09cd3231-8871-42cc-a1b2-f0826c378703\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:52.011165 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:52.011123 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6gm\" (UniqueName: \"kubernetes.io/projected/09cd3231-8871-42cc-a1b2-f0826c378703-kube-api-access-vc6gm\") pod \"postgres-69fb4594cf-pn29f\" (UID: \"09cd3231-8871-42cc-a1b2-f0826c378703\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:52.011386 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:52.011207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/09cd3231-8871-42cc-a1b2-f0826c378703-data\") pod \"postgres-69fb4594cf-pn29f\" (UID: \"09cd3231-8871-42cc-a1b2-f0826c378703\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:52.011577 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:52.011559 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/09cd3231-8871-42cc-a1b2-f0826c378703-data\") pod \"postgres-69fb4594cf-pn29f\" (UID: \"09cd3231-8871-42cc-a1b2-f0826c378703\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:52.023376 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:52.023345 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6gm\" (UniqueName: \"kubernetes.io/projected/09cd3231-8871-42cc-a1b2-f0826c378703-kube-api-access-vc6gm\") pod \"postgres-69fb4594cf-pn29f\" (UID: \"09cd3231-8871-42cc-a1b2-f0826c378703\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:52.138406 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:52.138310 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:52.279628 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:52.279582 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pn29f"] Jul 24 15:17:52.282323 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:17:52.282294 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cd3231_8871_42cc_a1b2_f0826c378703.slice/crio-7ec48a48e91e66b77c8b3fd2682ee4f33b9fdaa4e79c4ed4a73bcc9dd2245bef WatchSource:0}: Error finding container 7ec48a48e91e66b77c8b3fd2682ee4f33b9fdaa4e79c4ed4a73bcc9dd2245bef: Status 404 returned error can't find the container with id 7ec48a48e91e66b77c8b3fd2682ee4f33b9fdaa4e79c4ed4a73bcc9dd2245bef Jul 24 15:17:53.158643 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:53.158588 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" event={"ID":"09cd3231-8871-42cc-a1b2-f0826c378703","Type":"ContainerStarted","Data":"7ec48a48e91e66b77c8b3fd2682ee4f33b9fdaa4e79c4ed4a73bcc9dd2245bef"} Jul 24 15:17:58.191530 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:58.191483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" event={"ID":"09cd3231-8871-42cc-a1b2-f0826c378703","Type":"ContainerStarted","Data":"d992bcee98f3bce0542211dcfe54ccdcc035409e53a885c400b06db514023ee5"} Jul 24 15:17:58.191985 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:58.191611 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:17:58.210424 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:17:58.210368 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" podStartSLOduration=2.3539513530000002 podStartE2EDuration="7.210350429s" totalImagesPullingTime="4.856399076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:17:58.207024785 +0000 UTC m=+592.293676699" watchObservedRunningTime="2026-07-24 15:17:58.210350429 +0000 UTC m=+592.297002408" Jul 24 15:18:04.228419 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:04.228385 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pn29f" Jul 24 15:18:04.584793 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:04.584746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:04.792287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:04.792237 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rw4lv"] Jul 24 15:18:04.860468 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:04.860376 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rw4lv"] Jul 24 15:18:04.860468 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:04.860401 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:04.927092 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:04.927047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlhf\" (UniqueName: \"kubernetes.io/projected/d870b073-b4e7-4961-98d8-645ac65509c4-kube-api-access-tqlhf\") pod \"authorino-5d4b67c446-rw4lv\" (UID: \"d870b073-b4e7-4961-98d8-645ac65509c4\") " pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:05.027839 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:05.027798 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlhf\" (UniqueName: \"kubernetes.io/projected/d870b073-b4e7-4961-98d8-645ac65509c4-kube-api-access-tqlhf\") pod \"authorino-5d4b67c446-rw4lv\" (UID: \"d870b073-b4e7-4961-98d8-645ac65509c4\") " pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:05.044158 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:05.044121 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:05.045281 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:05.045229 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlhf\" (UniqueName: \"kubernetes.io/projected/d870b073-b4e7-4961-98d8-645ac65509c4-kube-api-access-tqlhf\") pod \"authorino-5d4b67c446-rw4lv\" (UID: \"d870b073-b4e7-4961-98d8-645ac65509c4\") " pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:05.060747 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:05.060713 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rw4lv"] Jul 24 15:18:05.060999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:05.060986 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:05.236261 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:05.236214 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rw4lv"] Jul 24 15:18:05.238809 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:18:05.238781 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd870b073_b4e7_4961_98d8_645ac65509c4.slice/crio-b7d547be0c4a7eaa0ef32b58f2dc7ba07cc2a2eb46897aa1f52eb19f8f88d35c WatchSource:0}: Error finding container b7d547be0c4a7eaa0ef32b58f2dc7ba07cc2a2eb46897aa1f52eb19f8f88d35c: Status 404 returned error can't find the container with id b7d547be0c4a7eaa0ef32b58f2dc7ba07cc2a2eb46897aa1f52eb19f8f88d35c Jul 24 15:18:06.245134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.245047 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" event={"ID":"d870b073-b4e7-4961-98d8-645ac65509c4","Type":"ContainerStarted","Data":"088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3"} Jul 24 15:18:06.245134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.245087 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" event={"ID":"d870b073-b4e7-4961-98d8-645ac65509c4","Type":"ContainerStarted","Data":"b7d547be0c4a7eaa0ef32b58f2dc7ba07cc2a2eb46897aa1f52eb19f8f88d35c"} Jul 24 15:18:06.245625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.245124 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" podUID="d870b073-b4e7-4961-98d8-645ac65509c4" containerName="authorino" containerID="cri-o://088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3" gracePeriod=30 Jul 24 15:18:06.263080 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.263027 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" podStartSLOduration=1.5413320289999999 podStartE2EDuration="2.263008142s" totalImagesPullingTime="721.676113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:06.26153371 +0000 UTC m=+600.348185623" watchObservedRunningTime="2026-07-24 15:18:06.263008142 +0000 UTC m=+600.349660055" Jul 24 15:18:06.357599 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.357573 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:18:06.358287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.358265 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:18:06.558563 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.558526 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:06.641712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.641678 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqlhf\" (UniqueName: \"kubernetes.io/projected/d870b073-b4e7-4961-98d8-645ac65509c4-kube-api-access-tqlhf\") pod \"d870b073-b4e7-4961-98d8-645ac65509c4\" (UID: \"d870b073-b4e7-4961-98d8-645ac65509c4\") " Jul 24 15:18:06.643819 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.643789 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d870b073-b4e7-4961-98d8-645ac65509c4-kube-api-access-tqlhf" (OuterVolumeSpecName: "kube-api-access-tqlhf") pod "d870b073-b4e7-4961-98d8-645ac65509c4" (UID: "d870b073-b4e7-4961-98d8-645ac65509c4"). InnerVolumeSpecName "kube-api-access-tqlhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:18:06.743044 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:06.743000 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlhf\" (UniqueName: \"kubernetes.io/projected/d870b073-b4e7-4961-98d8-645ac65509c4-kube-api-access-tqlhf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:18:07.251533 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.251492 2563 generic.go:356] "Generic (PLEG): container finished" podID="d870b073-b4e7-4961-98d8-645ac65509c4" containerID="088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3" exitCode=0 Jul 24 15:18:07.251533 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.251536 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" Jul 24 15:18:07.252055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.251532 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" event={"ID":"d870b073-b4e7-4961-98d8-645ac65509c4","Type":"ContainerDied","Data":"088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3"} Jul 24 15:18:07.252055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.251582 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-rw4lv" event={"ID":"d870b073-b4e7-4961-98d8-645ac65509c4","Type":"ContainerDied","Data":"b7d547be0c4a7eaa0ef32b58f2dc7ba07cc2a2eb46897aa1f52eb19f8f88d35c"} Jul 24 15:18:07.252055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.251604 2563 scope.go:117] "RemoveContainer" containerID="088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3" Jul 24 15:18:07.260780 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.260761 2563 scope.go:117] "RemoveContainer" containerID="088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3" Jul 24 15:18:07.261080 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:07.261061 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3\": container with ID starting with 088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3 not found: ID does not exist" containerID="088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3" Jul 24 15:18:07.261121 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.261092 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3"} err="failed to get container status \"088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3\": rpc error: code = NotFound desc = could not find container \"088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3\": container with ID starting with 088c2a520c2fcb2252707161d05c85f8d5a5f1663757b8725c72c6fe7f2d5af3 not found: ID does not exist" Jul 24 15:18:07.276489 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.276460 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rw4lv"] Jul 24 15:18:07.281995 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.281967 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-rw4lv"] Jul 24 15:18:07.330570 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.330527 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tlmgk"] Jul 24 15:18:07.330790 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.330761 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-tlmgk" podUID="75560229-c689-4fbe-a0b6-2cbdaeab83a5" containerName="authorino" containerID="cri-o://95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0" gracePeriod=30 Jul 24 15:18:07.586428 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.586399 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:18:07.650907 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.650861 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqxfj\" (UniqueName: \"kubernetes.io/projected/75560229-c689-4fbe-a0b6-2cbdaeab83a5-kube-api-access-qqxfj\") pod \"75560229-c689-4fbe-a0b6-2cbdaeab83a5\" (UID: \"75560229-c689-4fbe-a0b6-2cbdaeab83a5\") " Jul 24 15:18:07.652932 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.652900 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75560229-c689-4fbe-a0b6-2cbdaeab83a5-kube-api-access-qqxfj" (OuterVolumeSpecName: "kube-api-access-qqxfj") pod "75560229-c689-4fbe-a0b6-2cbdaeab83a5" (UID: "75560229-c689-4fbe-a0b6-2cbdaeab83a5"). InnerVolumeSpecName "kube-api-access-qqxfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:18:07.751712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:07.751660 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqxfj\" (UniqueName: \"kubernetes.io/projected/75560229-c689-4fbe-a0b6-2cbdaeab83a5-kube-api-access-qqxfj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:18:08.257968 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.257934 2563 generic.go:356] "Generic (PLEG): container finished" podID="75560229-c689-4fbe-a0b6-2cbdaeab83a5" containerID="95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0" exitCode=0 Jul 24 15:18:08.257968 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.257969 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tlmgk" Jul 24 15:18:08.258491 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.258009 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tlmgk" event={"ID":"75560229-c689-4fbe-a0b6-2cbdaeab83a5","Type":"ContainerDied","Data":"95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0"} Jul 24 15:18:08.258491 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.258052 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tlmgk" event={"ID":"75560229-c689-4fbe-a0b6-2cbdaeab83a5","Type":"ContainerDied","Data":"56be7d9c5b700e13c660aa11f65314ddbdad7082575a4b7d0f04bb533ccb7563"} Jul 24 15:18:08.258491 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.258074 2563 scope.go:117] "RemoveContainer" containerID="95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0" Jul 24 15:18:08.268349 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.268327 2563 scope.go:117] "RemoveContainer" containerID="95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0" Jul 24 15:18:08.268743 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:08.268672 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0\": container with ID starting with 95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0 not found: ID does not exist" containerID="95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0" Jul 24 15:18:08.268743 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.268708 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0"} err="failed to get container status \"95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0\": rpc error: code = NotFound desc = could not find container \"95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0\": container with ID starting with 95f57f42aa0c71a810dadab0d29e593fd7efdcfad7345f3eff72a063ec7c04a0 not found: ID does not exist" Jul 24 15:18:08.281919 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.281887 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tlmgk"] Jul 24 15:18:08.286065 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.286038 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tlmgk"] Jul 24 15:18:08.436826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.436791 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75560229-c689-4fbe-a0b6-2cbdaeab83a5" path="/var/lib/kubelet/pods/75560229-c689-4fbe-a0b6-2cbdaeab83a5/volumes" Jul 24 15:18:08.437132 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:08.437118 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d870b073-b4e7-4961-98d8-645ac65509c4" path="/var/lib/kubelet/pods/d870b073-b4e7-4961-98d8-645ac65509c4/volumes" Jul 24 15:18:12.260299 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.260232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:12.284100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284065 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-648fddf844-pl69r"] Jul 24 15:18:12.284467 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284452 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d870b073-b4e7-4961-98d8-645ac65509c4" containerName="authorino" Jul 24 15:18:12.284514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284469 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d870b073-b4e7-4961-98d8-645ac65509c4" containerName="authorino" Jul 24 15:18:12.284514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284497 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75560229-c689-4fbe-a0b6-2cbdaeab83a5" containerName="authorino" Jul 24 15:18:12.284514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284503 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="75560229-c689-4fbe-a0b6-2cbdaeab83a5" containerName="authorino" Jul 24 15:18:12.284609 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284555 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d870b073-b4e7-4961-98d8-645ac65509c4" containerName="authorino" Jul 24 15:18:12.284609 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.284568 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="75560229-c689-4fbe-a0b6-2cbdaeab83a5" containerName="authorino" Jul 24 15:18:12.287222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.287197 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:12.289864 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.289841 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:18:12.290021 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.289843 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:18:12.290021 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.289970 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gzt58\"" Jul 24 15:18:12.298201 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.298175 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-648fddf844-pl69r"] Jul 24 15:18:12.358444 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.358406 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:12.391535 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.391492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:12.391712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.391580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7cnz\" (UniqueName: \"kubernetes.io/projected/8a1160ee-5b15-4e58-aeeb-97abb0110b27-kube-api-access-q7cnz\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:12.459146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.459109 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:12.463996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.463966 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-549b6877d-tlqrn"] Jul 24 15:18:12.466473 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.466450 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:12.478023 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.477993 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549b6877d-tlqrn"] Jul 24 15:18:12.492334 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.492290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:12.492502 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.492369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7cnz\" (UniqueName: \"kubernetes.io/projected/8a1160ee-5b15-4e58-aeeb-97abb0110b27-kube-api-access-q7cnz\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:12.492502 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:12.492434 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 15:18:12.492589 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:12.492513 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert podName:8a1160ee-5b15-4e58-aeeb-97abb0110b27 nodeName:}" failed. No retries permitted until 2026-07-24 15:18:12.99249374 +0000 UTC m=+607.079145632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert") pod "maas-controller-648fddf844-pl69r" (UID: "8a1160ee-5b15-4e58-aeeb-97abb0110b27") : secret "maas-controller-webhook-cert" not found Jul 24 15:18:12.504021 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.503983 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7cnz\" (UniqueName: \"kubernetes.io/projected/8a1160ee-5b15-4e58-aeeb-97abb0110b27-kube-api-access-q7cnz\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:12.593182 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.593085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:12.593379 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.593189 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sv2g\" (UniqueName: \"kubernetes.io/projected/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-kube-api-access-4sv2g\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:12.694627 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.694586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sv2g\" (UniqueName: \"kubernetes.io/projected/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-kube-api-access-4sv2g\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:12.694831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.694692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:12.694831 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:12.694802 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 24 15:18:12.694913 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:12.694852 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert podName:7b129f3b-31d4-45b4-b5ff-16d0bdce2651 nodeName:}" failed. No retries permitted until 2026-07-24 15:18:13.194839532 +0000 UTC m=+607.281491424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert") pod "maas-controller-549b6877d-tlqrn" (UID: "7b129f3b-31d4-45b4-b5ff-16d0bdce2651") : secret "maas-controller-webhook-cert" not found Jul 24 15:18:12.706695 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.706645 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sv2g\" (UniqueName: \"kubernetes.io/projected/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-kube-api-access-4sv2g\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:12.809095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.809054 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:12.998592 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:12.998492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:13.000904 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.000874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert\") pod \"maas-controller-648fddf844-pl69r\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:13.198424 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.198385 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:13.200537 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.200385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:13.202793 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.202763 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert\") pod \"maas-controller-549b6877d-tlqrn\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:13.341515 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.341483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-648fddf844-pl69r"] Jul 24 15:18:13.343284 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:18:13.343222 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1160ee_5b15_4e58_aeeb_97abb0110b27.slice/crio-5071af63b05aa5559403b8724a5009b73bdbc3a3540de249a9d1384a6174cf7f WatchSource:0}: Error finding container 5071af63b05aa5559403b8724a5009b73bdbc3a3540de249a9d1384a6174cf7f: Status 404 returned error can't find the container with id 5071af63b05aa5559403b8724a5009b73bdbc3a3540de249a9d1384a6174cf7f Jul 24 15:18:13.377144 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.377104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:13.516095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:13.516067 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549b6877d-tlqrn"] Jul 24 15:18:13.517983 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:18:13.517955 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b129f3b_31d4_45b4_b5ff_16d0bdce2651.slice/crio-e8d0fea45ee5740b13d689702858ba3f94798cf7db5788d63e05627a993ee6c6 WatchSource:0}: Error finding container e8d0fea45ee5740b13d689702858ba3f94798cf7db5788d63e05627a993ee6c6: Status 404 returned error can't find the container with id e8d0fea45ee5740b13d689702858ba3f94798cf7db5788d63e05627a993ee6c6 Jul 24 15:18:14.303477 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:14.303431 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-648fddf844-pl69r" event={"ID":"8a1160ee-5b15-4e58-aeeb-97abb0110b27","Type":"ContainerStarted","Data":"5071af63b05aa5559403b8724a5009b73bdbc3a3540de249a9d1384a6174cf7f"} Jul 24 15:18:14.305720 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:14.305649 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-tlqrn" event={"ID":"7b129f3b-31d4-45b4-b5ff-16d0bdce2651","Type":"ContainerStarted","Data":"e8d0fea45ee5740b13d689702858ba3f94798cf7db5788d63e05627a993ee6c6"} Jul 24 15:18:16.216713 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.216657 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:16.318026 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.317980 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-tlqrn" event={"ID":"7b129f3b-31d4-45b4-b5ff-16d0bdce2651","Type":"ContainerStarted","Data":"9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6"} Jul 24 15:18:16.318207 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.318171 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:16.319609 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.319587 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-648fddf844-pl69r" event={"ID":"8a1160ee-5b15-4e58-aeeb-97abb0110b27","Type":"ContainerStarted","Data":"1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052"} Jul 24 15:18:16.319751 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.319627 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:16.349786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.349732 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-549b6877d-tlqrn" podStartSLOduration=1.8681915999999998 podStartE2EDuration="4.349713656s" totalImagesPullingTime="2.481522056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:16.348900856 +0000 UTC m=+610.435552764" watchObservedRunningTime="2026-07-24 15:18:16.349713656 +0000 UTC m=+610.436365570" Jul 24 15:18:16.350596 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.350566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:16.394407 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:16.394351 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-648fddf844-pl69r" podStartSLOduration=1.746929738 podStartE2EDuration="4.394333164s" totalImagesPullingTime="2.647403426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:16.393218257 +0000 UTC m=+610.479870169" watchObservedRunningTime="2026-07-24 15:18:16.394333164 +0000 UTC m=+610.480985081" Jul 24 15:18:27.327727 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.327691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:18:27.330170 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.327775 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:27.413133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.413084 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-648fddf844-pl69r"] Jul 24 15:18:27.413357 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.413283 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-648fddf844-pl69r" podUID="8a1160ee-5b15-4e58-aeeb-97abb0110b27" containerName="manager" containerID="cri-o://1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052" gracePeriod=30 Jul 24 15:18:27.666504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.666472 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:27.840122 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.840027 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert\") pod \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " Jul 24 15:18:27.840122 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.840106 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7cnz\" (UniqueName: \"kubernetes.io/projected/8a1160ee-5b15-4e58-aeeb-97abb0110b27-kube-api-access-q7cnz\") pod \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\" (UID: \"8a1160ee-5b15-4e58-aeeb-97abb0110b27\") " Jul 24 15:18:27.842332 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.842299 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8a1160ee-5b15-4e58-aeeb-97abb0110b27" (UID: "8a1160ee-5b15-4e58-aeeb-97abb0110b27"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:18:27.842460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.842372 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1160ee-5b15-4e58-aeeb-97abb0110b27-kube-api-access-q7cnz" (OuterVolumeSpecName: "kube-api-access-q7cnz") pod "8a1160ee-5b15-4e58-aeeb-97abb0110b27" (UID: "8a1160ee-5b15-4e58-aeeb-97abb0110b27"). InnerVolumeSpecName "kube-api-access-q7cnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:18:27.941560 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.941475 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a1160ee-5b15-4e58-aeeb-97abb0110b27-webhook-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:18:27.941741 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:27.941571 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7cnz\" (UniqueName: \"kubernetes.io/projected/8a1160ee-5b15-4e58-aeeb-97abb0110b27-kube-api-access-q7cnz\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:18:28.387474 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.387433 2563 generic.go:356] "Generic (PLEG): container finished" podID="8a1160ee-5b15-4e58-aeeb-97abb0110b27" containerID="1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052" exitCode=0 Jul 24 15:18:28.387957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.387498 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-648fddf844-pl69r" Jul 24 15:18:28.387957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.387529 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-648fddf844-pl69r" event={"ID":"8a1160ee-5b15-4e58-aeeb-97abb0110b27","Type":"ContainerDied","Data":"1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052"} Jul 24 15:18:28.387957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.387606 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-648fddf844-pl69r" event={"ID":"8a1160ee-5b15-4e58-aeeb-97abb0110b27","Type":"ContainerDied","Data":"5071af63b05aa5559403b8724a5009b73bdbc3a3540de249a9d1384a6174cf7f"} Jul 24 15:18:28.387957 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.387629 2563 scope.go:117] "RemoveContainer" containerID="1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052" Jul 24 15:18:28.397109 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.397088 2563 scope.go:117] "RemoveContainer" containerID="1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052" Jul 24 15:18:28.397473 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:18:28.397452 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052\": container with ID starting with 1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052 not found: ID does not exist" containerID="1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052" Jul 24 15:18:28.397547 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.397486 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052"} err="failed to get container status \"1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052\": rpc error: code = NotFound desc = could not find container \"1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052\": container with ID starting with 1331838b43521f349b82309d4263babf2940b103a5858f57791ee66ec7e55052 not found: ID does not exist" Jul 24 15:18:28.412847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.412809 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-648fddf844-pl69r"] Jul 24 15:18:28.415189 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.415161 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-648fddf844-pl69r"] Jul 24 15:18:28.437421 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:28.437387 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1160ee-5b15-4e58-aeeb-97abb0110b27" path="/var/lib/kubelet/pods/8a1160ee-5b15-4e58-aeeb-97abb0110b27/volumes" Jul 24 15:18:34.639529 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:34.639492 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:46.476585 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:46.476538 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:46.618503 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:46.618465 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:47.259627 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.259577 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:47.449048 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.449007 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59df598d7d-82bt4"] Jul 24 15:18:47.449396 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.449384 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1160ee-5b15-4e58-aeeb-97abb0110b27" containerName="manager" Jul 24 15:18:47.449451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.449398 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1160ee-5b15-4e58-aeeb-97abb0110b27" containerName="manager" Jul 24 15:18:47.449492 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.449462 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1160ee-5b15-4e58-aeeb-97abb0110b27" containerName="manager" Jul 24 15:18:47.455889 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.455859 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.458986 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.458955 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 15:18:47.459180 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.458959 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tkn9s\"" Jul 24 15:18:47.461286 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.461260 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59df598d7d-82bt4"] Jul 24 15:18:47.486711 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.486655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ff4f\" (UniqueName: \"kubernetes.io/projected/022cc6e5-502e-4932-8b4f-b19de9df2127-kube-api-access-9ff4f\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.487107 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.486745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cc6e5-502e-4932-8b4f-b19de9df2127-maas-api-tls\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.487107 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.486901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cc6e5-502e-4932-8b4f-b19de9df2127-openshift-service-ca\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.587744 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.587643 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cc6e5-502e-4932-8b4f-b19de9df2127-openshift-service-ca\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.587744 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.587701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff4f\" (UniqueName: \"kubernetes.io/projected/022cc6e5-502e-4932-8b4f-b19de9df2127-kube-api-access-9ff4f\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.587744 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.587735 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cc6e5-502e-4932-8b4f-b19de9df2127-maas-api-tls\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.588408 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.588376 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cc6e5-502e-4932-8b4f-b19de9df2127-openshift-service-ca\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.589855 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.589824 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:47.590420 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.590398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cc6e5-502e-4932-8b4f-b19de9df2127-maas-api-tls\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.596506 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.596486 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ff4f\" (UniqueName: \"kubernetes.io/projected/022cc6e5-502e-4932-8b4f-b19de9df2127-kube-api-access-9ff4f\") pod \"maas-api-59df598d7d-82bt4\" (UID: \"022cc6e5-502e-4932-8b4f-b19de9df2127\") " pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.768256 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.768201 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:47.881325 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.881282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:47.928452 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.928419 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59df598d7d-82bt4"] Jul 24 15:18:47.931153 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:18:47.931125 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022cc6e5_502e_4932_8b4f_b19de9df2127.slice/crio-fbb70114f24c1a0ac3061f423829991bf63298b0c79d5fc147b1e8d5aca3fca8 WatchSource:0}: Error finding container fbb70114f24c1a0ac3061f423829991bf63298b0c79d5fc147b1e8d5aca3fca8: Status 404 returned error can't find the container with id fbb70114f24c1a0ac3061f423829991bf63298b0c79d5fc147b1e8d5aca3fca8 Jul 24 15:18:47.932424 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.932404 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:18:47.977166 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.977127 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:47.988006 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.987977 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd"] Jul 24 15:18:47.992836 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.992814 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:47.995194 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.995150 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 15:18:47.995338 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:47.995320 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gpgk5\"" Jul 24 15:18:48.002645 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.002604 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd"] Jul 24 15:18:48.014938 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.014909 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hl4mb"] Jul 24 15:18:48.018538 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.018516 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.031966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.031934 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hl4mb"] Jul 24 15:18:48.092176 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.092138 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22baeaaa-964a-44be-bc00-89064290f48c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hl4mb\" (UID: \"22baeaaa-964a-44be-bc00-89064290f48c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.092402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.092200 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60911cd6-421b-4d37-8074-59fb448465ff-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-nx4jd\" (UID: \"60911cd6-421b-4d37-8074-59fb448465ff\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.092402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.092319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkgb\" (UniqueName: \"kubernetes.io/projected/60911cd6-421b-4d37-8074-59fb448465ff-kube-api-access-wgkgb\") pod \"payload-pre-processing-687dbb6cf6-nx4jd\" (UID: \"60911cd6-421b-4d37-8074-59fb448465ff\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.092402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.092356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnp5d\" (UniqueName: \"kubernetes.io/projected/22baeaaa-964a-44be-bc00-89064290f48c-kube-api-access-fnp5d\") pod \"payload-processing-5cc6cd444b-hl4mb\" (UID: \"22baeaaa-964a-44be-bc00-89064290f48c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.193217 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.193117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22baeaaa-964a-44be-bc00-89064290f48c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hl4mb\" (UID: \"22baeaaa-964a-44be-bc00-89064290f48c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.193217 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.193189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60911cd6-421b-4d37-8074-59fb448465ff-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-nx4jd\" (UID: \"60911cd6-421b-4d37-8074-59fb448465ff\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.193475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.193234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkgb\" (UniqueName: \"kubernetes.io/projected/60911cd6-421b-4d37-8074-59fb448465ff-kube-api-access-wgkgb\") pod \"payload-pre-processing-687dbb6cf6-nx4jd\" (UID: \"60911cd6-421b-4d37-8074-59fb448465ff\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.193475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.193293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnp5d\" (UniqueName: \"kubernetes.io/projected/22baeaaa-964a-44be-bc00-89064290f48c-kube-api-access-fnp5d\") pod \"payload-processing-5cc6cd444b-hl4mb\" (UID: \"22baeaaa-964a-44be-bc00-89064290f48c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.193844 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.193820 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22baeaaa-964a-44be-bc00-89064290f48c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-hl4mb\" (UID: \"22baeaaa-964a-44be-bc00-89064290f48c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.193915 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.193885 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60911cd6-421b-4d37-8074-59fb448465ff-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-nx4jd\" (UID: \"60911cd6-421b-4d37-8074-59fb448465ff\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.206560 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.206523 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkgb\" (UniqueName: \"kubernetes.io/projected/60911cd6-421b-4d37-8074-59fb448465ff-kube-api-access-wgkgb\") pod \"payload-pre-processing-687dbb6cf6-nx4jd\" (UID: \"60911cd6-421b-4d37-8074-59fb448465ff\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.213008 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.212971 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnp5d\" (UniqueName: \"kubernetes.io/projected/22baeaaa-964a-44be-bc00-89064290f48c-kube-api-access-fnp5d\") pod \"payload-processing-5cc6cd444b-hl4mb\" (UID: \"22baeaaa-964a-44be-bc00-89064290f48c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.304890 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.304855 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:48.334109 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.334075 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:48.499515 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.499466 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-hl4mb"] Jul 24 15:18:48.500386 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.500360 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" event={"ID":"022cc6e5-502e-4932-8b4f-b19de9df2127","Type":"ContainerStarted","Data":"fbb70114f24c1a0ac3061f423829991bf63298b0c79d5fc147b1e8d5aca3fca8"} Jul 24 15:18:48.501401 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:18:48.501379 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22baeaaa_964a_44be_bc00_89064290f48c.slice/crio-0517eacc287d31a2e6817c10ef6febde4fd44c78d754d762b07bd7b1e93913e0 WatchSource:0}: Error finding container 0517eacc287d31a2e6817c10ef6febde4fd44c78d754d762b07bd7b1e93913e0: Status 404 returned error can't find the container with id 0517eacc287d31a2e6817c10ef6febde4fd44c78d754d762b07bd7b1e93913e0 Jul 24 15:18:48.679267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:48.676439 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd"] Jul 24 15:18:49.535425 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:49.535377 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" event={"ID":"22baeaaa-964a-44be-bc00-89064290f48c","Type":"ContainerStarted","Data":"0517eacc287d31a2e6817c10ef6febde4fd44c78d754d762b07bd7b1e93913e0"} Jul 24 15:18:49.538854 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:49.538764 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" event={"ID":"60911cd6-421b-4d37-8074-59fb448465ff","Type":"ContainerStarted","Data":"205e53e5dd5fff3bf9e3608031972bae971d9e7713f76653f7a47be0cbb1eebf"} Jul 24 15:18:50.547220 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:50.547177 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" event={"ID":"022cc6e5-502e-4932-8b4f-b19de9df2127","Type":"ContainerStarted","Data":"f29bddc82598e589a95b0c84f60f328f9581d69c35affddf44b60885701047f1"} Jul 24 15:18:50.547683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:50.547541 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:50.571494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:50.569760 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" podStartSLOduration=1.317869266 podStartE2EDuration="3.56973439s" totalImagesPullingTime="2.251865124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:50.565722252 +0000 UTC m=+644.652374170" watchObservedRunningTime="2026-07-24 15:18:50.56973439 +0000 UTC m=+644.656386305" Jul 24 15:18:51.554965 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:51.554927 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" event={"ID":"22baeaaa-964a-44be-bc00-89064290f48c","Type":"ContainerStarted","Data":"89d4da54e54ae041aac1e7bac4efaee9eeb0f6abbed48482b99116a74ca28afa"} Jul 24 15:18:51.555536 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:51.555514 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:18:51.557049 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:51.557019 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" event={"ID":"60911cd6-421b-4d37-8074-59fb448465ff","Type":"ContainerStarted","Data":"583b0279a1d3191044a8207bd31fef36d17cfab4229c1e5c3219620e46c33c04"} Jul 24 15:18:51.557178 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:51.557126 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:18:51.576756 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:51.576697 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" podStartSLOduration=2.071230307 podStartE2EDuration="4.576672924s" totalImagesPullingTime="2.505442617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:51.57647508 +0000 UTC m=+645.663126994" watchObservedRunningTime="2026-07-24 15:18:51.576672924 +0000 UTC m=+645.663324826" Jul 24 15:18:51.596056 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:51.596005 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" podStartSLOduration=2.276181201 podStartE2EDuration="4.595985365s" totalImagesPullingTime="2.319804164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:18:51.594671069 +0000 UTC m=+645.681322980" watchObservedRunningTime="2026-07-24 15:18:51.595985365 +0000 UTC m=+645.682637277" Jul 24 15:18:55.594359 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:55.594275 2563 generic.go:356] "Generic (PLEG): container finished" podID="bf5060b3-a279-4958-af38-7bcff776112b" containerID="9d90d6578377d338eaa41fb53ef38f8427f9ba90009917c0a78aee4a578bbbf5" exitCode=0 Jul 24 15:18:55.594359 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:55.594321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" event={"ID":"bf5060b3-a279-4958-af38-7bcff776112b","Type":"ContainerDied","Data":"9d90d6578377d338eaa41fb53ef38f8427f9ba90009917c0a78aee4a578bbbf5"} Jul 24 15:18:55.594813 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:55.594785 2563 scope.go:117] "RemoveContainer" containerID="9d90d6578377d338eaa41fb53ef38f8427f9ba90009917c0a78aee4a578bbbf5" Jul 24 15:18:55.595832 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:55.595794 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:18:55.595911 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:55.595864 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:18:55.595911 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:55.595908 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:18:56.570769 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:56.570730 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59df598d7d-82bt4" Jul 24 15:18:56.604131 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:56.604090 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" event={"ID":"bf5060b3-a279-4958-af38-7bcff776112b","Type":"ContainerStarted","Data":"97cdf6ac8d1f0e34f39f63b1528249b34d768c39e474f5bb71ec0bc5be42e255"} Jul 24 15:18:57.341207 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:57.341166 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:18:57.342450 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:57.342421 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v8z9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 15:18:57.342575 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:57.342488 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" podUID="bf5060b3-a279-4958-af38-7bcff776112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 15:18:57.596356 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:57.596236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:57.688261 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:57.688208 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:18:58.341307 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:58.341273 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v8z9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 15:18:58.341483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:58.341328 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" podUID="bf5060b3-a279-4958-af38-7bcff776112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 15:18:59.341298 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:59.341230 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v8z9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 15:18:59.341703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:18:59.341321 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" podUID="bf5060b3-a279-4958-af38-7bcff776112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 15:19:00.007558 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:00.007521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:19:00.341625 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:00.341581 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-v8z9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 15:19:00.342129 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:00.341647 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" podUID="bf5060b3-a279-4958-af38-7bcff776112b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 15:19:01.345267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:01.345208 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:19:01.346395 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:01.346367 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-v8z9f" Jul 24 15:19:02.567287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:02.567237 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-nx4jd" Jul 24 15:19:03.575820 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:03.575789 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-hl4mb" Jul 24 15:19:17.628523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.628486 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8"] Jul 24 15:19:17.634523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.634500 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.638147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.638116 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t7tsm\"" Jul 24 15:19:17.638329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.638166 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:19:17.638329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.638203 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 15:19:17.638329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.638128 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:19:17.648900 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.648851 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8"] Jul 24 15:19:17.676480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.676446 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:19:17.769502 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.769461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.769502 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.769506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.769801 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.769575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.769801 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.769612 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.769801 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.769662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphpb\" (UniqueName: \"kubernetes.io/projected/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-kube-api-access-wphpb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.769801 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.769730 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870320 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphpb\" (UniqueName: \"kubernetes.io/projected/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-kube-api-access-wphpb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870885 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870849 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870885 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870862 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.870982 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.870908 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.872787 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.872756 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.873177 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.873157 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.881100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.881036 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphpb\" (UniqueName: \"kubernetes.io/projected/d992ce95-bdfa-4a82-8dbc-4709ecc2e500-kube-api-access-wphpb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8\" (UID: \"d992ce95-bdfa-4a82-8dbc-4709ecc2e500\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:17.947267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:17.947195 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:18.122937 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:18.122886 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8"] Jul 24 15:19:18.125109 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:19:18.125077 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd992ce95_bdfa_4a82_8dbc_4709ecc2e500.slice/crio-0a16ad109e5111e5cc1d5f51d5427cd60dd0d83dc826df228ee3c2efb370dbfd WatchSource:0}: Error finding container 0a16ad109e5111e5cc1d5f51d5427cd60dd0d83dc826df228ee3c2efb370dbfd: Status 404 returned error can't find the container with id 0a16ad109e5111e5cc1d5f51d5427cd60dd0d83dc826df228ee3c2efb370dbfd Jul 24 15:19:18.740073 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:18.740028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" event={"ID":"d992ce95-bdfa-4a82-8dbc-4709ecc2e500","Type":"ContainerStarted","Data":"0a16ad109e5111e5cc1d5f51d5427cd60dd0d83dc826df228ee3c2efb370dbfd"} Jul 24 15:19:19.747414 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:19.747377 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" event={"ID":"d992ce95-bdfa-4a82-8dbc-4709ecc2e500","Type":"ContainerStarted","Data":"7a37e263a8e9644a34d46092b391eaf09f325142794b49fc56274191f85e626d"} Jul 24 15:19:19.772712 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:19.772657 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" podStartSLOduration=1.719198343 podStartE2EDuration="2.772640293s" totalImagesPullingTime="1.05344195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:19:19.770726082 +0000 UTC m=+673.857378000" watchObservedRunningTime="2026-07-24 15:19:19.772640293 +0000 UTC m=+673.859292206" Jul 24 15:19:25.083216 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:25.083174 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:19:27.948053 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:27.947990 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:27.948612 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:27.948087 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:27.961346 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:27.961315 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:28.808410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:28.808379 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8" Jul 24 15:19:33.765293 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:33.765236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:19:37.622379 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.622338 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l"] Jul 24 15:19:37.626232 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.626205 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.628841 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.628818 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 15:19:37.636265 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.636222 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l"] Jul 24 15:19:37.647526 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.647493 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.647709 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.647547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.647709 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.647604 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d2c\" (UniqueName: \"kubernetes.io/projected/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-kube-api-access-f9d2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.647709 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.647642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.647709 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.647669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.647709 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.647689 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.656795 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.656760 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:19:37.748560 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.748517 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d2c\" (UniqueName: \"kubernetes.io/projected/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-kube-api-access-f9d2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.748746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.748574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.748746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.748601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.748746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.748627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.748746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.748655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.748746 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.748696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.749151 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.749130 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.749218 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.749178 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.749304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.749189 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.751003 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.750973 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.751303 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.751284 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.759068 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.759040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d2c\" (UniqueName: \"kubernetes.io/projected/a419ae67-0b6d-4616-a8d5-95dfc74b1b4b-kube-api-access-f9d2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l\" (UID: \"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:37.938948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:37.938842 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:38.086227 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:38.086199 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l"] Jul 24 15:19:38.087922 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:19:38.087881 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda419ae67_0b6d_4616_a8d5_95dfc74b1b4b.slice/crio-e89d797d24af5a1c19c94600cc124fb5afb6a76f10ea52b2f108a99e8884ab5c WatchSource:0}: Error finding container e89d797d24af5a1c19c94600cc124fb5afb6a76f10ea52b2f108a99e8884ab5c: Status 404 returned error can't find the container with id e89d797d24af5a1c19c94600cc124fb5afb6a76f10ea52b2f108a99e8884ab5c Jul 24 15:19:38.857949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:38.857899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" event={"ID":"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b","Type":"ContainerStarted","Data":"9c41d6677f17a4f75a439c5e736b3527181cdc4f5d721c00cd4a668f0ae8e0b7"} Jul 24 15:19:38.858463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:38.858010 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" event={"ID":"a419ae67-0b6d-4616-a8d5-95dfc74b1b4b","Type":"ContainerStarted","Data":"e89d797d24af5a1c19c94600cc124fb5afb6a76f10ea52b2f108a99e8884ab5c"} Jul 24 15:19:38.878090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:38.878029 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" podStartSLOduration=1.670990236 podStartE2EDuration="1.878011925s" totalImagesPullingTime="207.021689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:19:38.876028164 +0000 UTC m=+692.962680078" watchObservedRunningTime="2026-07-24 15:19:38.878011925 +0000 UTC m=+692.964663836" Jul 24 15:19:47.939358 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:47.939310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:47.939905 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:47.939385 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:47.952436 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:47.952403 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:48.824441 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.824407 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97"] Jul 24 15:19:48.828155 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.828135 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:48.830682 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.830652 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 24 15:19:48.837752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.837722 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97"] Jul 24 15:19:48.925524 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.925491 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l" Jul 24 15:19:48.928993 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.928957 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:19:48.947002 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.946966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:48.947518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.947038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ac9d62-1754-4271-965f-0b5480cd98ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:48.947518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.947060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:48.947518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.947099 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:48.947518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.947119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpgf\" (UniqueName: \"kubernetes.io/projected/32ac9d62-1754-4271-965f-0b5480cd98ad-kube-api-access-xhpgf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:48.947518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:48.947291 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048379 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048451 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ac9d62-1754-4271-965f-0b5480cd98ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048498 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048523 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpgf\" (UniqueName: \"kubernetes.io/projected/32ac9d62-1754-4271-965f-0b5480cd98ad-kube-api-access-xhpgf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048955 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048835 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048955 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.048955 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.048891 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.050873 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.050845 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32ac9d62-1754-4271-965f-0b5480cd98ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.051061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.051042 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32ac9d62-1754-4271-965f-0b5480cd98ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.057201 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.057169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpgf\" (UniqueName: \"kubernetes.io/projected/32ac9d62-1754-4271-965f-0b5480cd98ad-kube-api-access-xhpgf\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmh97\" (UID: \"32ac9d62-1754-4271-965f-0b5480cd98ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.140110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.140013 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:49.286015 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.285987 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97"] Jul 24 15:19:49.288086 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:19:49.288054 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ac9d62_1754_4271_965f_0b5480cd98ad.slice/crio-38cb344ad8010451c1ea84ead5252e112cf96e8f4e40c7e68c5befa345a3987b WatchSource:0}: Error finding container 38cb344ad8010451c1ea84ead5252e112cf96e8f4e40c7e68c5befa345a3987b: Status 404 returned error can't find the container with id 38cb344ad8010451c1ea84ead5252e112cf96e8f4e40c7e68c5befa345a3987b Jul 24 15:19:49.928317 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.928276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" event={"ID":"32ac9d62-1754-4271-965f-0b5480cd98ad","Type":"ContainerStarted","Data":"c687d9be636ef2355a17c39f31bccd5768a7673ec65595d23a6117b8fe56d1d4"} Jul 24 15:19:49.928545 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.928378 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" event={"ID":"32ac9d62-1754-4271-965f-0b5480cd98ad","Type":"ContainerStarted","Data":"38cb344ad8010451c1ea84ead5252e112cf96e8f4e40c7e68c5befa345a3987b"} Jul 24 15:19:49.952069 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:49.952012 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" podStartSLOduration=1.738696311 podStartE2EDuration="1.951992738s" totalImagesPullingTime="213.296427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:19:49.950031557 +0000 UTC m=+704.036683463" watchObservedRunningTime="2026-07-24 15:19:49.951992738 +0000 UTC m=+704.038644653" Jul 24 15:19:58.066629 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:58.066584 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:19:59.140887 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:59.140787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:59.140887 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:59.140872 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:59.153713 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:59.153683 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:19:59.997077 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:19:59.997045 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmh97" Jul 24 15:20:31.260137 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:20:31.260094 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:20:31.337850 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:20:31.337806 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:20:41.454216 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:20:41.454175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:20:41.555100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:20:41.555061 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:20:59.519865 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:20:59.519801 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:20:59.626800 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:20:59.626759 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:00.836851 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:00.836800 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:00.935466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:00.935430 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:03.124683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:03.124646 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:26.909367 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:26.909317 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:27.011068 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:27.011031 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:50.827260 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:50.827202 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-549b6877d-tlqrn"] Jul 24 15:21:50.827729 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:50.827424 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-549b6877d-tlqrn" podUID="7b129f3b-31d4-45b4-b5ff-16d0bdce2651" containerName="manager" containerID="cri-o://9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6" gracePeriod=30 Jul 24 15:21:51.079926 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.079834 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:21:51.195125 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.195074 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert\") pod \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " Jul 24 15:21:51.195344 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.195197 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sv2g\" (UniqueName: \"kubernetes.io/projected/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-kube-api-access-4sv2g\") pod \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\" (UID: \"7b129f3b-31d4-45b4-b5ff-16d0bdce2651\") " Jul 24 15:21:51.197322 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.197279 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-kube-api-access-4sv2g" (OuterVolumeSpecName: "kube-api-access-4sv2g") pod "7b129f3b-31d4-45b4-b5ff-16d0bdce2651" (UID: "7b129f3b-31d4-45b4-b5ff-16d0bdce2651"). InnerVolumeSpecName "kube-api-access-4sv2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:21:51.197466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.197357 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b129f3b-31d4-45b4-b5ff-16d0bdce2651" (UID: "7b129f3b-31d4-45b4-b5ff-16d0bdce2651"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:21:51.296425 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.296381 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sv2g\" (UniqueName: \"kubernetes.io/projected/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-kube-api-access-4sv2g\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:21:51.296425 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.296419 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b129f3b-31d4-45b4-b5ff-16d0bdce2651-webhook-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:21:51.620396 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.620362 2563 generic.go:356] "Generic (PLEG): container finished" podID="7b129f3b-31d4-45b4-b5ff-16d0bdce2651" containerID="9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6" exitCode=0 Jul 24 15:21:51.620593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.620409 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-tlqrn" Jul 24 15:21:51.620593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.620445 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-tlqrn" event={"ID":"7b129f3b-31d4-45b4-b5ff-16d0bdce2651","Type":"ContainerDied","Data":"9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6"} Jul 24 15:21:51.620593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.620496 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-tlqrn" event={"ID":"7b129f3b-31d4-45b4-b5ff-16d0bdce2651","Type":"ContainerDied","Data":"e8d0fea45ee5740b13d689702858ba3f94798cf7db5788d63e05627a993ee6c6"} Jul 24 15:21:51.620593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.620521 2563 scope.go:117] "RemoveContainer" containerID="9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6" Jul 24 15:21:51.630424 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.630404 2563 scope.go:117] "RemoveContainer" containerID="9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6" Jul 24 15:21:51.630745 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:21:51.630725 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6\": container with ID starting with 9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6 not found: ID does not exist" containerID="9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6" Jul 24 15:21:51.630804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.630755 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6"} err="failed to get container status \"9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6\": rpc error: code = NotFound desc = could not find container \"9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6\": container with ID starting with 9350d59d1ac9971b2f8782b601a4cb949091bfeda4dd591b92aacae9996e50c6 not found: ID does not exist" Jul 24 15:21:51.650741 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.650709 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-549b6877d-tlqrn"] Jul 24 15:21:51.657410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:51.657370 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-549b6877d-tlqrn"] Jul 24 15:21:52.436950 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:52.436916 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b129f3b-31d4-45b4-b5ff-16d0bdce2651" path="/var/lib/kubelet/pods/7b129f3b-31d4-45b4-b5ff-16d0bdce2651/volumes" Jul 24 15:21:56.100944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:56.100909 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:21:56.203714 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:21:56.203683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:22:52.496255 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.496214 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-549b6877d-4mdqd"] Jul 24 15:22:52.496739 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.496636 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b129f3b-31d4-45b4-b5ff-16d0bdce2651" containerName="manager" Jul 24 15:22:52.496739 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.496648 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b129f3b-31d4-45b4-b5ff-16d0bdce2651" containerName="manager" Jul 24 15:22:52.496739 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.496714 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b129f3b-31d4-45b4-b5ff-16d0bdce2651" containerName="manager" Jul 24 15:22:52.499829 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.499809 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.503173 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.503147 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:22:52.503173 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.503149 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gzt58\"" Jul 24 15:22:52.503406 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.503207 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:22:52.510740 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.510709 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549b6877d-4mdqd"] Jul 24 15:22:52.563120 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.563083 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpsf\" (UniqueName: \"kubernetes.io/projected/6e721420-6153-4dab-b537-dc8a83030029-kube-api-access-9lpsf\") pod \"maas-controller-549b6877d-4mdqd\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.563120 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.563130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e721420-6153-4dab-b537-dc8a83030029-webhook-cert\") pod \"maas-controller-549b6877d-4mdqd\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.664529 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.664494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpsf\" (UniqueName: \"kubernetes.io/projected/6e721420-6153-4dab-b537-dc8a83030029-kube-api-access-9lpsf\") pod \"maas-controller-549b6877d-4mdqd\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.664529 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.664541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e721420-6153-4dab-b537-dc8a83030029-webhook-cert\") pod \"maas-controller-549b6877d-4mdqd\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.667120 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.667089 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e721420-6153-4dab-b537-dc8a83030029-webhook-cert\") pod \"maas-controller-549b6877d-4mdqd\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.674896 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.674870 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpsf\" (UniqueName: \"kubernetes.io/projected/6e721420-6153-4dab-b537-dc8a83030029-kube-api-access-9lpsf\") pod \"maas-controller-549b6877d-4mdqd\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.838412 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.838368 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:52.985480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:52.985454 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549b6877d-4mdqd"] Jul 24 15:22:52.987646 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:22:52.987619 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e721420_6153_4dab_b537_dc8a83030029.slice/crio-2f43550e12c89fefca72412c6a1a90a06bb60bb8849ce3a6f32d0bdbaddb72da WatchSource:0}: Error finding container 2f43550e12c89fefca72412c6a1a90a06bb60bb8849ce3a6f32d0bdbaddb72da: Status 404 returned error can't find the container with id 2f43550e12c89fefca72412c6a1a90a06bb60bb8849ce3a6f32d0bdbaddb72da Jul 24 15:22:53.970653 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:53.970614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-4mdqd" event={"ID":"6e721420-6153-4dab-b537-dc8a83030029","Type":"ContainerStarted","Data":"bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02"} Jul 24 15:22:53.971063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:53.970706 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-4mdqd" event={"ID":"6e721420-6153-4dab-b537-dc8a83030029","Type":"ContainerStarted","Data":"2f43550e12c89fefca72412c6a1a90a06bb60bb8849ce3a6f32d0bdbaddb72da"} Jul 24 15:22:53.971063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:53.970730 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:22:53.989475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:22:53.989417 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-549b6877d-4mdqd" podStartSLOduration=1.551376661 podStartE2EDuration="1.989400984s" totalImagesPullingTime="438.024323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:22:53.988026465 +0000 UTC m=+888.074678380" watchObservedRunningTime="2026-07-24 15:22:53.989400984 +0000 UTC m=+888.076052896" Jul 24 15:23:04.979835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:04.979758 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:23:06.393914 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:06.393885 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:23:06.397552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:06.397522 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:23:09.553179 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:09.553135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:09.660032 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:09.659993 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-549b6877d-4mdqd"] Jul 24 15:23:09.660299 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:09.660238 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-549b6877d-4mdqd" podUID="6e721420-6153-4dab-b537-dc8a83030029" containerName="manager" containerID="cri-o://bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02" gracePeriod=30 Jul 24 15:23:09.917328 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:09.917292 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:23:10.024533 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.024496 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e721420-6153-4dab-b537-dc8a83030029-webhook-cert\") pod \"6e721420-6153-4dab-b537-dc8a83030029\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " Jul 24 15:23:10.024719 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.024588 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lpsf\" (UniqueName: \"kubernetes.io/projected/6e721420-6153-4dab-b537-dc8a83030029-kube-api-access-9lpsf\") pod \"6e721420-6153-4dab-b537-dc8a83030029\" (UID: \"6e721420-6153-4dab-b537-dc8a83030029\") " Jul 24 15:23:10.026614 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.026576 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e721420-6153-4dab-b537-dc8a83030029-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6e721420-6153-4dab-b537-dc8a83030029" (UID: "6e721420-6153-4dab-b537-dc8a83030029"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:23:10.026742 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.026699 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e721420-6153-4dab-b537-dc8a83030029-kube-api-access-9lpsf" (OuterVolumeSpecName: "kube-api-access-9lpsf") pod "6e721420-6153-4dab-b537-dc8a83030029" (UID: "6e721420-6153-4dab-b537-dc8a83030029"). InnerVolumeSpecName "kube-api-access-9lpsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:23:10.063642 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.063603 2563 generic.go:356] "Generic (PLEG): container finished" podID="6e721420-6153-4dab-b537-dc8a83030029" containerID="bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02" exitCode=0 Jul 24 15:23:10.063804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.063652 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-4mdqd" Jul 24 15:23:10.063804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.063685 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-4mdqd" event={"ID":"6e721420-6153-4dab-b537-dc8a83030029","Type":"ContainerDied","Data":"bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02"} Jul 24 15:23:10.063804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.063743 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-4mdqd" event={"ID":"6e721420-6153-4dab-b537-dc8a83030029","Type":"ContainerDied","Data":"2f43550e12c89fefca72412c6a1a90a06bb60bb8849ce3a6f32d0bdbaddb72da"} Jul 24 15:23:10.063804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.063767 2563 scope.go:117] "RemoveContainer" containerID="bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02" Jul 24 15:23:10.073645 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.073623 2563 scope.go:117] "RemoveContainer" containerID="bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02" Jul 24 15:23:10.073989 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:23:10.073966 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02\": container with ID starting with bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02 not found: ID does not exist" containerID="bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02" Jul 24 15:23:10.074053 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.073999 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02"} err="failed to get container status \"bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02\": rpc error: code = NotFound desc = could not find container \"bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02\": container with ID starting with bb3251717e9ddcd3b80642729b7de449f2721b8defda2b1517150de7eef57a02 not found: ID does not exist" Jul 24 15:23:10.087037 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.086951 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-549b6877d-4mdqd"] Jul 24 15:23:10.091390 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.091358 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-549b6877d-4mdqd"] Jul 24 15:23:10.125786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.125740 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lpsf\" (UniqueName: \"kubernetes.io/projected/6e721420-6153-4dab-b537-dc8a83030029-kube-api-access-9lpsf\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:23:10.125786 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.125780 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e721420-6153-4dab-b537-dc8a83030029-webhook-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:23:10.437110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.437030 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e721420-6153-4dab-b537-dc8a83030029" path="/var/lib/kubelet/pods/6e721420-6153-4dab-b537-dc8a83030029/volumes" Jul 24 15:23:10.801046 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.801005 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:10.914918 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.914876 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-549b6877d-f6dhb"] Jul 24 15:23:10.915322 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.915307 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e721420-6153-4dab-b537-dc8a83030029" containerName="manager" Jul 24 15:23:10.915378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.915324 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e721420-6153-4dab-b537-dc8a83030029" containerName="manager" Jul 24 15:23:10.915417 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.915396 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e721420-6153-4dab-b537-dc8a83030029" containerName="manager" Jul 24 15:23:10.920153 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.920125 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:10.923400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.923360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:23:10.923400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.923398 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:23:10.923635 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.923360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gzt58\"" Jul 24 15:23:10.930134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:10.930104 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549b6877d-f6dhb"] Jul 24 15:23:11.033685 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.033642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b954\" (UniqueName: \"kubernetes.io/projected/be2e7525-961f-4090-9321-d583b655b8a0-kube-api-access-6b954\") pod \"maas-controller-549b6877d-f6dhb\" (UID: \"be2e7525-961f-4090-9321-d583b655b8a0\") " pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.033869 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.033693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be2e7525-961f-4090-9321-d583b655b8a0-webhook-cert\") pod \"maas-controller-549b6877d-f6dhb\" (UID: \"be2e7525-961f-4090-9321-d583b655b8a0\") " pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.134909 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.134799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b954\" (UniqueName: \"kubernetes.io/projected/be2e7525-961f-4090-9321-d583b655b8a0-kube-api-access-6b954\") pod \"maas-controller-549b6877d-f6dhb\" (UID: \"be2e7525-961f-4090-9321-d583b655b8a0\") " pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.134909 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.134858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be2e7525-961f-4090-9321-d583b655b8a0-webhook-cert\") pod \"maas-controller-549b6877d-f6dhb\" (UID: \"be2e7525-961f-4090-9321-d583b655b8a0\") " pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.137425 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.137398 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be2e7525-961f-4090-9321-d583b655b8a0-webhook-cert\") pod \"maas-controller-549b6877d-f6dhb\" (UID: \"be2e7525-961f-4090-9321-d583b655b8a0\") " pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.143601 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.143573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b954\" (UniqueName: \"kubernetes.io/projected/be2e7525-961f-4090-9321-d583b655b8a0-kube-api-access-6b954\") pod \"maas-controller-549b6877d-f6dhb\" (UID: \"be2e7525-961f-4090-9321-d583b655b8a0\") " pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.232229 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.232173 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:11.378415 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:11.378389 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549b6877d-f6dhb"] Jul 24 15:23:11.380330 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:23:11.380302 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2e7525_961f_4090_9321_d583b655b8a0.slice/crio-b361f38eb18b15fe7de16469a540725362a3d5155d6ba9e796ed7cd98cf29c8c WatchSource:0}: Error finding container b361f38eb18b15fe7de16469a540725362a3d5155d6ba9e796ed7cd98cf29c8c: Status 404 returned error can't find the container with id b361f38eb18b15fe7de16469a540725362a3d5155d6ba9e796ed7cd98cf29c8c Jul 24 15:23:12.078683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:12.078574 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-f6dhb" event={"ID":"be2e7525-961f-4090-9321-d583b655b8a0","Type":"ContainerStarted","Data":"db7e0d944d01a2bb439c0ef812ed814b94cb23fcf9cacace8a46f917e3afb652"} Jul 24 15:23:12.078683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:12.078639 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549b6877d-f6dhb" event={"ID":"be2e7525-961f-4090-9321-d583b655b8a0","Type":"ContainerStarted","Data":"b361f38eb18b15fe7de16469a540725362a3d5155d6ba9e796ed7cd98cf29c8c"} Jul 24 15:23:12.078683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:12.078666 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:12.098835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:12.098779 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-549b6877d-f6dhb" podStartSLOduration=1.694774956 podStartE2EDuration="2.098762365s" totalImagesPullingTime="403.987409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:23:12.096033575 +0000 UTC m=+906.182685488" watchObservedRunningTime="2026-07-24 15:23:12.098762365 +0000 UTC m=+906.185414279" Jul 24 15:23:23.087117 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:23.087083 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-549b6877d-f6dhb" Jul 24 15:23:44.847031 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:44.846984 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:44.945155 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:44.945113 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:45.084019 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:45.083973 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:50.862184 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:50.862139 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:50.924947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:50.924910 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:51.014682 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:51.014637 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:51.115956 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:51.115850 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:51.291216 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:51.291177 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:57.132003 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:57.131962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:23:57.235354 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:23:57.235306 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:25:09.476082 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:25:09.476038 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:26:20.775233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:26:20.775146 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:26:40.433348 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:26:40.433298 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:26:40.503662 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:26:40.503618 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:26:49.916724 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:26:49.916660 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:27:06.904429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:27:06.904390 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:28:06.435497 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:28:06.435463 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:28:06.445100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:28:06.445070 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:31:43.355343 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:31:43.355304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:31:48.869363 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:31:48.869326 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:15.708918 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:15.708818 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:15.799566 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:15.799528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:26.477705 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:26.477661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:26.578949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:26.578907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:26.642706 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:26.642662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:28.039901 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:28.039854 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:29.156926 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:29.156877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:41.802370 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:41.802331 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:32:42.156905 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:32:42.156807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:33:00.178572 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:33:00.178521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:33:06.478607 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:33:06.478578 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:33:06.484609 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:33:06.484583 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:34:11.690531 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:34:11.690486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:35:36.369346 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:35:36.369305 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:35:58.174024 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:35:58.173989 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:01.493789 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:01.493748 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:01.592042 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:01.592004 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:01.986464 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:01.986429 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:08.226160 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:08.226115 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:08.326178 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:08.326135 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:08.469631 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:08.469592 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:08.822339 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:08.822292 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:09.168169 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:09.168054 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:10.916059 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:10.916017 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:11.021667 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.021628 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:11.040154 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.040115 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql"] Jul 24 15:36:11.043785 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.043754 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.047635 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.047612 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f8777f9-serving-cert\"" Jul 24 15:36:11.078617 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.078577 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql"] Jul 24 15:36:11.151765 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.151708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:11.180138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.180036 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.180138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.180089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-openshift-service-ca\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.180138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.180123 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgtkp\" (UniqueName: \"kubernetes.io/projected/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-kube-api-access-dgtkp\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.281001 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.280960 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.281227 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.281032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-openshift-service-ca\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.281227 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.281068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgtkp\" (UniqueName: \"kubernetes.io/projected/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-kube-api-access-dgtkp\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.281227 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:36:11.281119 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-serving-cert: secret "maas-api-e2e-ait-6f8777f9-serving-cert" not found Jul 24 15:36:11.281430 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:36:11.281232 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls podName:9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:11.781212227 +0000 UTC m=+1685.867864118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls") pod "maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" (UID: "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316") : secret "maas-api-e2e-ait-6f8777f9-serving-cert" not found Jul 24 15:36:11.281770 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.281747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-openshift-service-ca\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.293553 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.293524 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgtkp\" (UniqueName: \"kubernetes.io/projected/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-kube-api-access-dgtkp\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.424611 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.424574 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:11.777552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.777506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:11.785979 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.785943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.788797 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.788771 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls\") pod \"maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:11.874445 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.874410 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:11.954838 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:11.954806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:12.120815 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:12.120788 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql"] Jul 24 15:36:12.123658 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:36:12.123621 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6bb17f_1dc8_4ac1_85eb_3ab5c81eb316.slice/crio-3610ccbcde3be63095895b11ed1d8643d4b0f1c72ba1d836ae49e5dd525ba4fa WatchSource:0}: Error finding container 3610ccbcde3be63095895b11ed1d8643d4b0f1c72ba1d836ae49e5dd525ba4fa: Status 404 returned error can't find the container with id 3610ccbcde3be63095895b11ed1d8643d4b0f1c72ba1d836ae49e5dd525ba4fa Jul 24 15:36:12.124956 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:12.124936 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:36:12.564815 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:12.564780 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" event={"ID":"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316","Type":"ContainerStarted","Data":"3610ccbcde3be63095895b11ed1d8643d4b0f1c72ba1d836ae49e5dd525ba4fa"} Jul 24 15:36:13.574795 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:13.574758 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" event={"ID":"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316","Type":"ContainerStarted","Data":"8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d"} Jul 24 15:36:13.575287 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:13.574819 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:13.595866 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:13.595815 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" podStartSLOduration=2.175891543 podStartE2EDuration="2.595798315s" totalImagesPullingTime="419.906772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:13.595162362 +0000 UTC m=+1687.681814273" watchObservedRunningTime="2026-07-24 15:36:13.595798315 +0000 UTC m=+1687.682450229" Jul 24 15:36:19.586860 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:19.586831 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:47.295311 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.295269 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql"] Jul 24 15:36:47.295734 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.295503 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" podUID="9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" containerName="maas-api" containerID="cri-o://8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d" gracePeriod=30 Jul 24 15:36:47.338703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.338658 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:47.555255 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.555165 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:47.614442 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.614399 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgtkp\" (UniqueName: \"kubernetes.io/projected/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-kube-api-access-dgtkp\") pod \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " Jul 24 15:36:47.614651 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.614499 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls\") pod \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " Jul 24 15:36:47.614651 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.614552 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-openshift-service-ca\") pod \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\" (UID: \"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316\") " Jul 24 15:36:47.614949 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.614922 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" (UID: "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:36:47.616711 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.616686 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-kube-api-access-dgtkp" (OuterVolumeSpecName: "kube-api-access-dgtkp") pod "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" (UID: "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316"). InnerVolumeSpecName "kube-api-access-dgtkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:36:47.616778 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.616695 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" (UID: "9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:36:47.716414 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.716373 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:36:47.716414 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.716418 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:36:47.716636 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.716435 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgtkp\" (UniqueName: \"kubernetes.io/projected/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316-kube-api-access-dgtkp\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:36:47.781686 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.781650 2563 generic.go:356] "Generic (PLEG): container finished" podID="9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" containerID="8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d" exitCode=0 Jul 24 15:36:47.781870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.781697 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" Jul 24 15:36:47.781870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.781733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" event={"ID":"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316","Type":"ContainerDied","Data":"8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d"} Jul 24 15:36:47.781870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.781788 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql" event={"ID":"9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316","Type":"ContainerDied","Data":"3610ccbcde3be63095895b11ed1d8643d4b0f1c72ba1d836ae49e5dd525ba4fa"} Jul 24 15:36:47.781870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.781813 2563 scope.go:117] "RemoveContainer" containerID="8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d" Jul 24 15:36:47.792831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.792805 2563 scope.go:117] "RemoveContainer" containerID="8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d" Jul 24 15:36:47.793148 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:36:47.793126 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d\": container with ID starting with 8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d not found: ID does not exist" containerID="8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d" Jul 24 15:36:47.793209 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.793156 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d"} err="failed to get container status \"8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d\": rpc error: code = NotFound desc = could not find container \"8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d\": container with ID starting with 8ae6b6573b1a02762d25fd0aebbb94588f81ac926ddc306dcd96c9ad4cb81c3d not found: ID does not exist" Jul 24 15:36:47.810074 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.809987 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql"] Jul 24 15:36:47.817693 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:47.817661 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f8777f9-5c697c8d6c-7llql"] Jul 24 15:36:48.437511 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:48.437477 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" path="/var/lib/kubelet/pods/9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316/volumes" Jul 24 15:36:48.924394 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:48.924354 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:51.670207 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:51.670168 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:51.873394 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:51.873354 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:52.101061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:52.101026 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:52.416460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:52.416361 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:52.523463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:52.523421 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:52.646513 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:52.646479 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:53.023392 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:53.023349 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:53.298673 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:53.298577 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:55.286055 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:55.286010 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:55.503824 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:55.503787 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:56.182322 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:56.182280 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:36:56.278737 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:36:56.278701 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:20.303473 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.303428 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6"] Jul 24 15:37:20.304069 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.304047 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" containerName="maas-api" Jul 24 15:37:20.304155 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.304072 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" containerName="maas-api" Jul 24 15:37:20.304208 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.304178 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6bb17f-1dc8-4ac1-85eb-3ab5c81eb316" containerName="maas-api" Jul 24 15:37:20.307366 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.307343 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:20.310496 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.310475 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:37:20.318476 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.318448 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6"] Jul 24 15:37:20.326774 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.326715 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc8b986-a46b-4bc1-8390-9cb1aed639cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6\" (UID: \"4dc8b986-a46b-4bc1-8390-9cb1aed639cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:20.427367 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.427322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc8b986-a46b-4bc1-8390-9cb1aed639cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6\" (UID: \"4dc8b986-a46b-4bc1-8390-9cb1aed639cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:20.428040 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.428009 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc8b986-a46b-4bc1-8390-9cb1aed639cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6\" (UID: \"4dc8b986-a46b-4bc1-8390-9cb1aed639cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:20.619355 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.619261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:20.771215 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.771187 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6"] Jul 24 15:37:20.773665 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:37:20.773632 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc8b986_a46b_4bc1_8390_9cb1aed639cc.slice/crio-f35e80342e98a27f452cc034f069065b478a5ea3599fc317e1630aefa8d97cff WatchSource:0}: Error finding container f35e80342e98a27f452cc034f069065b478a5ea3599fc317e1630aefa8d97cff: Status 404 returned error can't find the container with id f35e80342e98a27f452cc034f069065b478a5ea3599fc317e1630aefa8d97cff Jul 24 15:37:20.971872 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:20.971786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" event={"ID":"4dc8b986-a46b-4bc1-8390-9cb1aed639cc","Type":"ContainerStarted","Data":"f35e80342e98a27f452cc034f069065b478a5ea3599fc317e1630aefa8d97cff"} Jul 24 15:37:23.991653 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:23.991560 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" event={"ID":"4dc8b986-a46b-4bc1-8390-9cb1aed639cc","Type":"ContainerStarted","Data":"f676433d426ddf0f2d672308c92cddaff36f54972c324750753aca807b3ec38e"} Jul 24 15:37:24.016231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:24.016171 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" podStartSLOduration=1.443461294 podStartE2EDuration="4.016150614s" totalImagesPullingTime="2.57268932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:24.014605386 +0000 UTC m=+1758.101257298" watchObservedRunningTime="2026-07-24 15:37:24.016150614 +0000 UTC m=+1758.102802529" Jul 24 15:37:24.999418 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:24.999374 2563 generic.go:356] "Generic (PLEG): container finished" podID="4dc8b986-a46b-4bc1-8390-9cb1aed639cc" containerID="f676433d426ddf0f2d672308c92cddaff36f54972c324750753aca807b3ec38e" exitCode=0 Jul 24 15:37:24.999795 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:24.999429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" event={"ID":"4dc8b986-a46b-4bc1-8390-9cb1aed639cc","Type":"ContainerDied","Data":"f676433d426ddf0f2d672308c92cddaff36f54972c324750753aca807b3ec38e"} Jul 24 15:37:26.142397 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:26.142364 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:26.169595 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:26.169566 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc8b986-a46b-4bc1-8390-9cb1aed639cc-maas-api-service-ca\") pod \"4dc8b986-a46b-4bc1-8390-9cb1aed639cc\" (UID: \"4dc8b986-a46b-4bc1-8390-9cb1aed639cc\") " Jul 24 15:37:26.169936 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:26.169910 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc8b986-a46b-4bc1-8390-9cb1aed639cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4dc8b986-a46b-4bc1-8390-9cb1aed639cc" (UID: "4dc8b986-a46b-4bc1-8390-9cb1aed639cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:37:26.270495 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:26.270464 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dc8b986-a46b-4bc1-8390-9cb1aed639cc-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:37:27.013018 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:27.012985 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" Jul 24 15:37:27.013235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:27.013021 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6" event={"ID":"4dc8b986-a46b-4bc1-8390-9cb1aed639cc","Type":"ContainerDied","Data":"f35e80342e98a27f452cc034f069065b478a5ea3599fc317e1630aefa8d97cff"} Jul 24 15:37:27.013235 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:27.013070 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f35e80342e98a27f452cc034f069065b478a5ea3599fc317e1630aefa8d97cff" Jul 24 15:37:30.959274 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:30.959202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:31.040539 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:31.040500 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:31.119662 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:31.119627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:35.408608 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:35.408569 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6"] Jul 24 15:37:35.413021 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:35.412988 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-70e683b2-9qpz6"] Jul 24 15:37:35.419345 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:35.419310 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:35.604319 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:35.604272 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:35.850784 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:35.850746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:36.180262 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:36.180145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:36.439418 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:36.439334 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc8b986-a46b-4bc1-8390-9cb1aed639cc" path="/var/lib/kubelet/pods/4dc8b986-a46b-4bc1-8390-9cb1aed639cc/volumes" Jul 24 15:37:36.492003 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:36.491966 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:37.007159 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:37.007111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:39.331641 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:39.331586 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:39.447541 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:39.447509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:39.605458 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:39.605360 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:40.168567 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:40.168529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:40.269908 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:40.269871 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:37:40.463082 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:37:40.462995 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:05.161568 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.161525 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk"] Jul 24 15:38:05.161992 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.161977 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc8b986-a46b-4bc1-8390-9cb1aed639cc" containerName="revoke-keys" Jul 24 15:38:05.162038 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.161993 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc8b986-a46b-4bc1-8390-9cb1aed639cc" containerName="revoke-keys" Jul 24 15:38:05.162072 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.162064 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc8b986-a46b-4bc1-8390-9cb1aed639cc" containerName="revoke-keys" Jul 24 15:38:05.165226 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.165207 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:05.168043 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.168020 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:38:05.174316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.174289 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk"] Jul 24 15:38:05.329922 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.329875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42592960-83fb-44c7-9266-348fe814bc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk\" (UID: \"42592960-83fb-44c7-9266-348fe814bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:05.430798 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.430672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42592960-83fb-44c7-9266-348fe814bc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk\" (UID: \"42592960-83fb-44c7-9266-348fe814bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:05.431416 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.431394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42592960-83fb-44c7-9266-348fe814bc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk\" (UID: \"42592960-83fb-44c7-9266-348fe814bc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:05.480807 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.480766 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:05.626776 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:05.626725 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk"] Jul 24 15:38:05.628918 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:38:05.628890 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42592960_83fb_44c7_9266_348fe814bc95.slice/crio-36803ca776f82673ec177e2f92815391aa428ee0db7e1b3dbc593d0680fa6956 WatchSource:0}: Error finding container 36803ca776f82673ec177e2f92815391aa428ee0db7e1b3dbc593d0680fa6956: Status 404 returned error can't find the container with id 36803ca776f82673ec177e2f92815391aa428ee0db7e1b3dbc593d0680fa6956 Jul 24 15:38:06.236383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:06.236290 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" event={"ID":"42592960-83fb-44c7-9266-348fe814bc95","Type":"ContainerStarted","Data":"7ebcff6c3e13fa1b6db3061186268d6601c5583fb81598f9891f46f74290d67a"} Jul 24 15:38:06.236383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:06.236347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" event={"ID":"42592960-83fb-44c7-9266-348fe814bc95","Type":"ContainerStarted","Data":"36803ca776f82673ec177e2f92815391aa428ee0db7e1b3dbc593d0680fa6956"} Jul 24 15:38:06.254832 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:06.254764 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" podStartSLOduration=1.254743164 podStartE2EDuration="1.254743164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:06.252498025 +0000 UTC m=+1800.339149940" watchObservedRunningTime="2026-07-24 15:38:06.254743164 +0000 UTC m=+1800.341395079" Jul 24 15:38:06.517026 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:06.516994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:38:06.525114 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:06.525089 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:38:07.250977 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:07.250939 2563 generic.go:356] "Generic (PLEG): container finished" podID="42592960-83fb-44c7-9266-348fe814bc95" containerID="7ebcff6c3e13fa1b6db3061186268d6601c5583fb81598f9891f46f74290d67a" exitCode=0 Jul 24 15:38:07.251413 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:07.250993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" event={"ID":"42592960-83fb-44c7-9266-348fe814bc95","Type":"ContainerDied","Data":"7ebcff6c3e13fa1b6db3061186268d6601c5583fb81598f9891f46f74290d67a"} Jul 24 15:38:08.392304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:08.392276 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:08.558711 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:08.558669 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42592960-83fb-44c7-9266-348fe814bc95-maas-api-service-ca\") pod \"42592960-83fb-44c7-9266-348fe814bc95\" (UID: \"42592960-83fb-44c7-9266-348fe814bc95\") " Jul 24 15:38:08.559049 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:08.559013 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42592960-83fb-44c7-9266-348fe814bc95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42592960-83fb-44c7-9266-348fe814bc95" (UID: "42592960-83fb-44c7-9266-348fe814bc95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:08.659656 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:08.659616 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42592960-83fb-44c7-9266-348fe814bc95-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:38:09.264004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:09.263972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" event={"ID":"42592960-83fb-44c7-9266-348fe814bc95","Type":"ContainerDied","Data":"36803ca776f82673ec177e2f92815391aa428ee0db7e1b3dbc593d0680fa6956"} Jul 24 15:38:09.264161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:09.264014 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk" Jul 24 15:38:09.264161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:09.264021 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36803ca776f82673ec177e2f92815391aa428ee0db7e1b3dbc593d0680fa6956" Jul 24 15:38:15.977460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:15.977419 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:16.076571 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:16.076531 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:20.239161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:20.239119 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk"] Jul 24 15:38:20.243305 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:20.243268 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:20.245154 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:20.245130 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26f33d00-kt7rk"] Jul 24 15:38:20.436947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:20.436910 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42592960-83fb-44c7-9266-348fe814bc95" path="/var/lib/kubelet/pods/42592960-83fb-44c7-9266-348fe814bc95/volumes" Jul 24 15:38:21.446804 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:21.446764 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:21.666152 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:21.666119 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:21.969903 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:21.969864 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:22.265216 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:22.265175 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:22.601718 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:22.601627 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:24.556128 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:24.556085 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:24.676569 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:24.676533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:24.788988 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:24.788941 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:24.982608 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:24.982503 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:25.435820 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:25.435779 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:25.534151 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:25.534117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:30.237986 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:30.237945 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:38:49.417118 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.417076 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r"] Jul 24 15:38:49.417661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.417553 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42592960-83fb-44c7-9266-348fe814bc95" containerName="revoke-keys" Jul 24 15:38:49.417661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.417565 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42592960-83fb-44c7-9266-348fe814bc95" containerName="revoke-keys" Jul 24 15:38:49.417661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.417624 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42592960-83fb-44c7-9266-348fe814bc95" containerName="revoke-keys" Jul 24 15:38:49.420749 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.420724 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:49.427753 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.427719 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:38:49.441798 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.441759 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r"] Jul 24 15:38:49.541754 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.541710 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159cbaaa-2bff-4bc4-a6b0-24e3282729d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r\" (UID: \"159cbaaa-2bff-4bc4-a6b0-24e3282729d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:49.643132 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.643091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159cbaaa-2bff-4bc4-a6b0-24e3282729d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r\" (UID: \"159cbaaa-2bff-4bc4-a6b0-24e3282729d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:49.643796 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.643774 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159cbaaa-2bff-4bc4-a6b0-24e3282729d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r\" (UID: \"159cbaaa-2bff-4bc4-a6b0-24e3282729d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:49.731090 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.730994 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:49.889905 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:49.889874 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r"] Jul 24 15:38:49.891571 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:38:49.891543 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159cbaaa_2bff_4bc4_a6b0_24e3282729d5.slice/crio-de355dc5343edde545611d67071418ab1b4f0c52ba41ce6ff70a592791f24f56 WatchSource:0}: Error finding container de355dc5343edde545611d67071418ab1b4f0c52ba41ce6ff70a592791f24f56: Status 404 returned error can't find the container with id de355dc5343edde545611d67071418ab1b4f0c52ba41ce6ff70a592791f24f56 Jul 24 15:38:50.524100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:50.524063 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" event={"ID":"159cbaaa-2bff-4bc4-a6b0-24e3282729d5","Type":"ContainerStarted","Data":"8ebde003948e7ad3abc1b4fbc59466adce7ed0f63f9bb3e0a26b952635921d46"} Jul 24 15:38:50.524545 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:50.524116 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" event={"ID":"159cbaaa-2bff-4bc4-a6b0-24e3282729d5","Type":"ContainerStarted","Data":"de355dc5343edde545611d67071418ab1b4f0c52ba41ce6ff70a592791f24f56"} Jul 24 15:38:50.557160 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:50.557105 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" podStartSLOduration=1.557086951 podStartE2EDuration="1.557086951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:50.556437323 +0000 UTC m=+1844.643089236" watchObservedRunningTime="2026-07-24 15:38:50.557086951 +0000 UTC m=+1844.643738863" Jul 24 15:38:51.531909 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:51.531869 2563 generic.go:356] "Generic (PLEG): container finished" podID="159cbaaa-2bff-4bc4-a6b0-24e3282729d5" containerID="8ebde003948e7ad3abc1b4fbc59466adce7ed0f63f9bb3e0a26b952635921d46" exitCode=0 Jul 24 15:38:51.532402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:51.531929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" event={"ID":"159cbaaa-2bff-4bc4-a6b0-24e3282729d5","Type":"ContainerDied","Data":"8ebde003948e7ad3abc1b4fbc59466adce7ed0f63f9bb3e0a26b952635921d46"} Jul 24 15:38:52.694637 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:52.694604 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:52.776577 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:52.776532 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159cbaaa-2bff-4bc4-a6b0-24e3282729d5-maas-api-service-ca\") pod \"159cbaaa-2bff-4bc4-a6b0-24e3282729d5\" (UID: \"159cbaaa-2bff-4bc4-a6b0-24e3282729d5\") " Jul 24 15:38:52.776857 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:52.776832 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159cbaaa-2bff-4bc4-a6b0-24e3282729d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "159cbaaa-2bff-4bc4-a6b0-24e3282729d5" (UID: "159cbaaa-2bff-4bc4-a6b0-24e3282729d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:52.877929 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:52.877833 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/159cbaaa-2bff-4bc4-a6b0-24e3282729d5-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:38:53.545400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:53.545372 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" Jul 24 15:38:53.545400 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:53.545392 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r" event={"ID":"159cbaaa-2bff-4bc4-a6b0-24e3282729d5","Type":"ContainerDied","Data":"de355dc5343edde545611d67071418ab1b4f0c52ba41ce6ff70a592791f24f56"} Jul 24 15:38:53.545617 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:38:53.545437 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de355dc5343edde545611d67071418ab1b4f0c52ba41ce6ff70a592791f24f56" Jul 24 15:39:00.269573 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:00.269531 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:00.375480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:00.375435 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:00.484847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:00.484775 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:04.545476 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:04.545439 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:04.551363 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:04.551323 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r"] Jul 24 15:39:04.556319 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:04.556285 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-65c49f07-x8s4r"] Jul 24 15:39:04.751477 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:04.751438 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:05.084678 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:05.084642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:05.491466 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:05.491370 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:05.600149 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:05.600111 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:06.436876 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:06.436838 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="159cbaaa-2bff-4bc4-a6b0-24e3282729d5" path="/var/lib/kubelet/pods/159cbaaa-2bff-4bc4-a6b0-24e3282729d5/volumes" Jul 24 15:39:07.244180 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:07.244142 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:14.228655 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:14.228622 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:14.469084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:14.469049 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:14.659148 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:14.659114 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:15.191753 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:15.191708 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:41.203385 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:41.203347 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:49.215670 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:49.215633 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:49.474626 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:49.474529 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:39:49.675640 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:39:49.675592 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:05.491693 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:05.491637 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:11.679182 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:11.679130 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:11.790110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:11.790076 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:12.072464 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:12.072428 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:22.461479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:22.461440 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:23.283598 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:23.283566 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:30.746731 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:30.746683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:30.857675 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:30.857642 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:31.013556 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:31.013521 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:31.181857 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:31.181816 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:31.893071 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:31.893027 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:31.985955 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:31.985911 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:32.086083 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:32.086042 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:33.762577 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:33.762533 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:38.993828 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:38.993778 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:40.222154 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:40.222117 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:40.330398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:40.330363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:40.911884 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:40.911843 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:46.792701 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:46.792661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:46.872989 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:46.872947 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:47.082234 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:47.082138 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:47.483398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:47.483300 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:48.003169 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:48.003129 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:50.804469 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:50.804428 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:50.902549 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:50.902511 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:51.083075 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:51.082982 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:51.615463 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:51.615428 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:51.788231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:51.788187 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:51.888860 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:51.888768 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:55.314531 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:55.314496 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:56.024806 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:56.024762 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:56.123020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:56.122980 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:40:56.408352 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:40:56.408236 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:02.401676 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.401588 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:02.512835 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.512797 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:02.757397 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.757308 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw"] Jul 24 15:41:02.757764 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.757751 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159cbaaa-2bff-4bc4-a6b0-24e3282729d5" containerName="revoke-keys" Jul 24 15:41:02.757811 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.757765 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="159cbaaa-2bff-4bc4-a6b0-24e3282729d5" containerName="revoke-keys" Jul 24 15:41:02.757851 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.757823 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="159cbaaa-2bff-4bc4-a6b0-24e3282729d5" containerName="revoke-keys" Jul 24 15:41:02.761139 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.761117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:02.764170 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.764143 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:41:02.769560 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.769530 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw"] Jul 24 15:41:02.865775 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.865731 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95c88d9-549f-4ec5-a42a-0b253fef7566-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw\" (UID: \"b95c88d9-549f-4ec5-a42a-0b253fef7566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:02.966699 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.966658 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95c88d9-549f-4ec5-a42a-0b253fef7566-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw\" (UID: \"b95c88d9-549f-4ec5-a42a-0b253fef7566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:02.967426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:02.967404 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95c88d9-549f-4ec5-a42a-0b253fef7566-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw\" (UID: \"b95c88d9-549f-4ec5-a42a-0b253fef7566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:03.073267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:03.073209 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:03.221166 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:03.221135 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw"] Jul 24 15:41:03.223701 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:41:03.223664 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95c88d9_549f_4ec5_a42a_0b253fef7566.slice/crio-01371cc9d208db11467786e7138fc4c9795f37d43749e70d0fc3d6f916daccb4 WatchSource:0}: Error finding container 01371cc9d208db11467786e7138fc4c9795f37d43749e70d0fc3d6f916daccb4: Status 404 returned error can't find the container with id 01371cc9d208db11467786e7138fc4c9795f37d43749e70d0fc3d6f916daccb4 Jul 24 15:41:03.307722 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:03.307682 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" event={"ID":"b95c88d9-549f-4ec5-a42a-0b253fef7566","Type":"ContainerStarted","Data":"01371cc9d208db11467786e7138fc4c9795f37d43749e70d0fc3d6f916daccb4"} Jul 24 15:41:04.315716 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:04.315680 2563 generic.go:356] "Generic (PLEG): container finished" podID="b95c88d9-549f-4ec5-a42a-0b253fef7566" containerID="725e813829c9ebc7b7dea172816d4eb25f0192c14d0f422bc10c0fad656cd24e" exitCode=0 Jul 24 15:41:04.316143 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:04.315767 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" event={"ID":"b95c88d9-549f-4ec5-a42a-0b253fef7566","Type":"ContainerDied","Data":"725e813829c9ebc7b7dea172816d4eb25f0192c14d0f422bc10c0fad656cd24e"} Jul 24 15:41:05.466604 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:05.466570 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:05.591607 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:05.591493 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95c88d9-549f-4ec5-a42a-0b253fef7566-maas-api-service-ca\") pod \"b95c88d9-549f-4ec5-a42a-0b253fef7566\" (UID: \"b95c88d9-549f-4ec5-a42a-0b253fef7566\") " Jul 24 15:41:05.591862 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:05.591836 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95c88d9-549f-4ec5-a42a-0b253fef7566-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b95c88d9-549f-4ec5-a42a-0b253fef7566" (UID: "b95c88d9-549f-4ec5-a42a-0b253fef7566"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:41:05.591947 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:05.591901 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b95c88d9-549f-4ec5-a42a-0b253fef7566-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:41:06.329398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:06.329366 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" Jul 24 15:41:06.329398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:06.329381 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw" event={"ID":"b95c88d9-549f-4ec5-a42a-0b253fef7566","Type":"ContainerDied","Data":"01371cc9d208db11467786e7138fc4c9795f37d43749e70d0fc3d6f916daccb4"} Jul 24 15:41:06.329619 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:06.329422 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01371cc9d208db11467786e7138fc4c9795f37d43749e70d0fc3d6f916daccb4" Jul 24 15:41:13.432809 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:13.432766 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:13.531421 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:13.531373 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:13.628803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:13.628769 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:17.855189 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:17.855152 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw"] Jul 24 15:41:17.863977 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:17.863921 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a586f80f-2rjbw"] Jul 24 15:41:17.870594 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:17.870560 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:18.254358 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:18.254232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:18.437200 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:18.437163 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b95c88d9-549f-4ec5-a42a-0b253fef7566" path="/var/lib/kubelet/pods/b95c88d9-549f-4ec5-a42a-0b253fef7566/volumes" Jul 24 15:41:24.607499 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:24.607459 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:24.764927 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:24.764891 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:24.896036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:24.895906 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:25.125585 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:25.125540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:25.247634 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:25.247536 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:25.348568 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:25.348516 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:25.475972 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:25.475896 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:31.596905 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:31.596861 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:31.700857 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:31.700821 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:31.801364 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:31.801324 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:31.982279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:31.982133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:38.300345 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:38.300304 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:38.402133 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:38.402095 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:45.057460 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:45.057418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:45.418748 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:45.418645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:45.668945 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:45.668853 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:48.079563 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:48.079514 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:48.176448 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:48.176404 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:48.274805 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:48.274746 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:48.567661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:48.567625 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:48.953839 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:48.953740 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:50.313033 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:50.312995 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:52.150497 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:52.150452 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:52.534669 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:52.534625 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:56.782516 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:56.782472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:41:56.857163 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:41:56.857116 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:42:07.860765 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:42:07.860724 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:42:47.621788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:42:47.621750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:42:47.725751 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:42:47.725707 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:42:47.829191 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:42:47.829148 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:42:52.142689 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:42:52.142648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:42:55.359997 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:42:55.359962 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:00.430968 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:00.430929 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:06.558562 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:06.558530 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:43:06.568828 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:06.568803 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:43:36.696549 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:36.696491 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:37.842971 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:37.842928 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:43.679146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:43.679102 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:43.787453 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:43.787415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:43.989187 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:43.989093 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:44.232311 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:44.232274 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:45.291928 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:45.291886 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:45.387504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:45.387462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:45.513689 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:45.513656 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:46.289381 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:46.289342 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:46.390989 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:46.390952 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:47.869776 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:47.869737 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:53.912134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:53.912089 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:54.029965 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:54.029929 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:43:54.603999 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:43:54.603946 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:00.887329 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:00.887211 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:01.001641 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:01.001596 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:01.104811 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:01.104777 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:06.696051 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:06.696011 2563 scope.go:117] "RemoveContainer" containerID="f676433d426ddf0f2d672308c92cddaff36f54972c324750753aca807b3ec38e" Jul 24 15:44:06.705295 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:06.705272 2563 scope.go:117] "RemoveContainer" containerID="7ebcff6c3e13fa1b6db3061186268d6601c5583fb81598f9891f46f74290d67a" Jul 24 15:44:11.658349 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:11.658312 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:11.968309 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:11.968217 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:32.941757 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:32.941714 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:39.986996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:39.986951 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:40.077267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:40.077202 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:40.249982 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:40.249877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:40.645144 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:40.645098 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:40.938669 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:40.938568 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:43.875295 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:43.875252 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:43.993794 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:43.993750 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:44.080568 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:44.080524 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:44.192576 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:44.192478 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:44.853752 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:44.853709 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:44.955615 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:44.955574 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:48.425633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:48.425597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:48.720144 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:48.720028 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:49.098983 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:49.098940 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:49.291571 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:49.291528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:51.880826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:51.880785 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:51.978216 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:51.978176 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:52.151954 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:52.151837 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:52.477149 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:52.477059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:52.766519 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:52.766483 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:52.862116 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:52.862076 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:44:56.385421 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:44:56.385381 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:45:04.588639 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:45:04.588598 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:45:06.733571 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:45:06.733537 2563 scope.go:117] "RemoveContainer" containerID="8ebde003948e7ad3abc1b4fbc59466adce7ed0f63f9bb3e0a26b952635921d46" Jul 24 15:45:08.642118 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:45:08.642065 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:45:42.348530 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:45:42.348486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:45:49.542389 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:45:49.542348 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:20.979970 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:20.979901 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:21.855665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:21.855607 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:32.423073 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:32.423040 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:32.491116 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:32.491070 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:32.589847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:32.589804 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:37.051660 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:37.051623 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:37.438172 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:37.438073 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:43.390166 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:43.390118 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:43.459616 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:43.459579 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:43.730163 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:43.730080 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:43.889644 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:43.889606 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:54.446405 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:54.446369 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:56.062325 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:56.062228 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:59.096674 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:59.096581 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:46:59.477576 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:46:59.477450 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:05.279544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:05.279501 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:05.440521 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:05.440472 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:05.554516 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:05.554428 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:05.780161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:05.780126 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:06.119919 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:06.119879 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:06.438480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:06.438377 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:06.563398 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:06.563359 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:06.764383 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:06.764343 2563 scope.go:117] "RemoveContainer" containerID="725e813829c9ebc7b7dea172816d4eb25f0192c14d0f422bc10c0fad656cd24e" Jul 24 15:47:09.567862 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:09.567723 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:09.653420 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:09.653375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:09.734964 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:09.734925 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:09.974659 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:09.974571 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:10.410442 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:10.410405 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:10.504310 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:10.504269 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:14.045868 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:14.045823 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:14.546300 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:14.546261 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:14.819923 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:14.819811 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:14.948710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:14.948674 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:17.591942 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:17.591888 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:17.699544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:17.699502 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:17.981784 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:17.981678 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:18.432569 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:18.432527 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:18.533620 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:18.533582 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:18.710960 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:18.710857 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:22.089470 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:22.089419 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:45.797567 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:45.797498 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:45.899893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:45.899854 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:46.000261 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:46.000203 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:50.100862 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:50.100823 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:50.491523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:50.491427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:56.448152 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:56.448110 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:56.548575 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:56.548525 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:56.698313 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:56.698207 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:47:56.909639 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:47:56.909598 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:03.935340 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:03.935290 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:06.596213 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:06.596183 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:48:06.607107 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:06.607077 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:48:12.482800 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:12.482754 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:12.556542 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:12.556504 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:12.722681 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:12.722647 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:17.111725 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:17.111677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:17.509646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:17.509608 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:23.348849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:23.348806 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:23.449452 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:23.449418 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:23.611215 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:23.611113 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:23.845539 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:23.845480 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:24.196108 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:24.196077 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:24.509498 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:24.509468 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:24.693405 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:24.693375 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:27.023894 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:27.023855 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:27.103314 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:27.103274 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:27.234903 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:27.234853 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:27.580815 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:27.580779 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:27.803571 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:27.803540 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:27.891148 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:27.891059 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:28.173165 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:28.173079 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:31.481435 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:31.481397 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:31.834770 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:31.834731 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:32.131487 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:32.131382 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:32.248710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:32.248669 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:35.120208 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:35.120165 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:35.219888 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:35.219846 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:35.346717 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:35.346671 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:35.450915 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:35.450810 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:35.849654 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:35.849619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:35.958519 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:35.958486 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:39.575280 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:39.575224 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:48:53.141767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:48:53.141726 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:49:20.324806 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:49:20.324752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:49:22.007726 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:49:22.007686 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:49:40.286404 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:49:40.286365 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:49:40.387410 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:49:40.387372 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:51:28.222865 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:51:28.222829 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:51:29.343110 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:51:29.343019 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:51:49.656558 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:51:49.656515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:52:11.663676 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:52:11.663637 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:52:56.037710 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:52:56.037675 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:52:56.864931 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:52:56.864882 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:53:06.633073 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:53:06.632994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:53:06.646668 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:53:06.646641 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:53:08.470593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:53:08.470528 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:53:43.845142 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:53:43.845105 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:26.088832 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:26.088792 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:27.033402 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.033363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:27.934951 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.934907 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7966-140546768140976"] Jul 24 15:54:27.935401 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.935385 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b95c88d9-549f-4ec5-a42a-0b253fef7566" containerName="revoke-keys" Jul 24 15:54:27.935452 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.935403 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b95c88d9-549f-4ec5-a42a-0b253fef7566" containerName="revoke-keys" Jul 24 15:54:27.935491 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.935469 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b95c88d9-549f-4ec5-a42a-0b253fef7566" containerName="revoke-keys" Jul 24 15:54:27.944272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.939106 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:27.944272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.942349 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zfjqm\"" Jul 24 15:54:27.949260 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:27.949216 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768140976"] Jul 24 15:54:28.046831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:28.046783 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8d8r\" (UniqueName: \"kubernetes.io/projected/e6beeff6-d90a-4667-81c3-a42300a6fd2b-kube-api-access-w8d8r\") pod \"test-curl-7966-140546768140976\" (UID: \"e6beeff6-d90a-4667-81c3-a42300a6fd2b\") " pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:28.147623 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:28.147578 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8d8r\" (UniqueName: \"kubernetes.io/projected/e6beeff6-d90a-4667-81c3-a42300a6fd2b-kube-api-access-w8d8r\") pod \"test-curl-7966-140546768140976\" (UID: \"e6beeff6-d90a-4667-81c3-a42300a6fd2b\") " pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:28.156469 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:28.156437 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8d8r\" (UniqueName: \"kubernetes.io/projected/e6beeff6-d90a-4667-81c3-a42300a6fd2b-kube-api-access-w8d8r\") pod \"test-curl-7966-140546768140976\" (UID: \"e6beeff6-d90a-4667-81c3-a42300a6fd2b\") " pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:28.256434 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:28.256337 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:28.400499 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:28.400469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768140976"] Jul 24 15:54:28.402889 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:28.402855 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6beeff6_d90a_4667_81c3_a42300a6fd2b.slice/crio-86d9432d66524598fc386abb44f258a52c02e0e0a2789114ce5cf16bb97b38f8 WatchSource:0}: Error finding container 86d9432d66524598fc386abb44f258a52c02e0e0a2789114ce5cf16bb97b38f8: Status 404 returned error can't find the container with id 86d9432d66524598fc386abb44f258a52c02e0e0a2789114ce5cf16bb97b38f8 Jul 24 15:54:28.404101 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:28.404083 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:54:29.010759 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:29.010641 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768140976" event={"ID":"e6beeff6-d90a-4667-81c3-a42300a6fd2b","Type":"ContainerStarted","Data":"86d9432d66524598fc386abb44f258a52c02e0e0a2789114ce5cf16bb97b38f8"} Jul 24 15:54:30.021833 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:30.021795 2563 generic.go:356] "Generic (PLEG): container finished" podID="e6beeff6-d90a-4667-81c3-a42300a6fd2b" containerID="90d403b416d4089b54e43916c156ca262ca4f48010cf9f8c942567e270451c7f" exitCode=0 Jul 24 15:54:30.022335 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:30.021882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768140976" event={"ID":"e6beeff6-d90a-4667-81c3-a42300a6fd2b","Type":"ContainerDied","Data":"90d403b416d4089b54e43916c156ca262ca4f48010cf9f8c942567e270451c7f"} Jul 24 15:54:31.168268 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.168220 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:31.190281 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.190234 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7966-140546768140976_e6beeff6-d90a-4667-81c3-a42300a6fd2b/test-curl-7966-140546768140976/0.log" Jul 24 15:54:31.213936 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.213900 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7966-140546768140976"] Jul 24 15:54:31.216333 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.216308 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7966-140546768140976"] Jul 24 15:54:31.278518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.278423 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8d8r\" (UniqueName: \"kubernetes.io/projected/e6beeff6-d90a-4667-81c3-a42300a6fd2b-kube-api-access-w8d8r\") pod \"e6beeff6-d90a-4667-81c3-a42300a6fd2b\" (UID: \"e6beeff6-d90a-4667-81c3-a42300a6fd2b\") " Jul 24 15:54:31.280544 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.280504 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6beeff6-d90a-4667-81c3-a42300a6fd2b-kube-api-access-w8d8r" (OuterVolumeSpecName: "kube-api-access-w8d8r") pod "e6beeff6-d90a-4667-81c3-a42300a6fd2b" (UID: "e6beeff6-d90a-4667-81c3-a42300a6fd2b"). InnerVolumeSpecName "kube-api-access-w8d8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:31.334910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.329625 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7966-140546768141104"] Jul 24 15:54:31.334910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.330030 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6beeff6-d90a-4667-81c3-a42300a6fd2b" containerName="test-curl-7966-140546768140976" Jul 24 15:54:31.334910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.330046 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6beeff6-d90a-4667-81c3-a42300a6fd2b" containerName="test-curl-7966-140546768140976" Jul 24 15:54:31.334910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.330117 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6beeff6-d90a-4667-81c3-a42300a6fd2b" containerName="test-curl-7966-140546768140976" Jul 24 15:54:31.334910 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.333200 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:31.340178 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.340147 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768141104"] Jul 24 15:54:31.379932 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.379896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq9c\" (UniqueName: \"kubernetes.io/projected/953175da-3c79-4f99-a3b2-094491ec0cb6-kube-api-access-bwq9c\") pod \"test-curl-7966-140546768141104\" (UID: \"953175da-3c79-4f99-a3b2-094491ec0cb6\") " pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:31.380117 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.380014 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8d8r\" (UniqueName: \"kubernetes.io/projected/e6beeff6-d90a-4667-81c3-a42300a6fd2b-kube-api-access-w8d8r\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:54:31.480505 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.480456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq9c\" (UniqueName: \"kubernetes.io/projected/953175da-3c79-4f99-a3b2-094491ec0cb6-kube-api-access-bwq9c\") pod \"test-curl-7966-140546768141104\" (UID: \"953175da-3c79-4f99-a3b2-094491ec0cb6\") " pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:31.488797 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.488760 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq9c\" (UniqueName: \"kubernetes.io/projected/953175da-3c79-4f99-a3b2-094491ec0cb6-kube-api-access-bwq9c\") pod \"test-curl-7966-140546768141104\" (UID: \"953175da-3c79-4f99-a3b2-094491ec0cb6\") " pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:31.644826 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.644731 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:31.787483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:31.787457 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768141104"] Jul 24 15:54:31.789800 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:31.789771 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod953175da_3c79_4f99_a3b2_094491ec0cb6.slice/crio-a198cb9dfb9e5cc5ee63d4574dd1310d9111fbd47b1bffb3e232aa8e12f2bbd4 WatchSource:0}: Error finding container a198cb9dfb9e5cc5ee63d4574dd1310d9111fbd47b1bffb3e232aa8e12f2bbd4: Status 404 returned error can't find the container with id a198cb9dfb9e5cc5ee63d4574dd1310d9111fbd47b1bffb3e232aa8e12f2bbd4 Jul 24 15:54:32.035435 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:32.035396 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768141104" event={"ID":"953175da-3c79-4f99-a3b2-094491ec0cb6","Type":"ContainerStarted","Data":"a198cb9dfb9e5cc5ee63d4574dd1310d9111fbd47b1bffb3e232aa8e12f2bbd4"} Jul 24 15:54:32.036681 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:32.036659 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86d9432d66524598fc386abb44f258a52c02e0e0a2789114ce5cf16bb97b38f8" Jul 24 15:54:32.036681 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:32.036673 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:32.047878 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:54:32.047843 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7966-140546768140976\" is forbidden: User \"system:node:ip-10-0-132-115.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-115.ec2.internal' and this object" podUID="e6beeff6-d90a-4667-81c3-a42300a6fd2b" pod="opendatahub/test-curl-7966-140546768140976" Jul 24 15:54:32.443801 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:32.438042 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6beeff6-d90a-4667-81c3-a42300a6fd2b" path="/var/lib/kubelet/pods/e6beeff6-d90a-4667-81c3-a42300a6fd2b/volumes" Jul 24 15:54:33.044578 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:33.044543 2563 generic.go:356] "Generic (PLEG): container finished" podID="953175da-3c79-4f99-a3b2-094491ec0cb6" containerID="1902b69ba12a3a3314d654d85252db930aa9775f300be3b92f9c06ce8331f9d7" exitCode=0 Jul 24 15:54:33.044741 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:33.044628 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768141104" event={"ID":"953175da-3c79-4f99-a3b2-094491ec0cb6","Type":"ContainerDied","Data":"1902b69ba12a3a3314d654d85252db930aa9775f300be3b92f9c06ce8331f9d7"} Jul 24 15:54:34.194775 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.194745 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:34.214477 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.214449 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7966-140546768141104_953175da-3c79-4f99-a3b2-094491ec0cb6/test-curl-7966-140546768141104/0.log" Jul 24 15:54:34.236545 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.236511 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7966-140546768141104"] Jul 24 15:54:34.240475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.240442 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7966-140546768141104"] Jul 24 15:54:34.306973 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.306875 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwq9c\" (UniqueName: \"kubernetes.io/projected/953175da-3c79-4f99-a3b2-094491ec0cb6-kube-api-access-bwq9c\") pod \"953175da-3c79-4f99-a3b2-094491ec0cb6\" (UID: \"953175da-3c79-4f99-a3b2-094491ec0cb6\") " Jul 24 15:54:34.309078 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.309039 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953175da-3c79-4f99-a3b2-094491ec0cb6-kube-api-access-bwq9c" (OuterVolumeSpecName: "kube-api-access-bwq9c") pod "953175da-3c79-4f99-a3b2-094491ec0cb6" (UID: "953175da-3c79-4f99-a3b2-094491ec0cb6"). InnerVolumeSpecName "kube-api-access-bwq9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:34.391222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.391187 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:34.391620 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.391606 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953175da-3c79-4f99-a3b2-094491ec0cb6" containerName="test-curl-7966-140546768141104" Jul 24 15:54:34.391665 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.391622 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="953175da-3c79-4f99-a3b2-094491ec0cb6" containerName="test-curl-7966-140546768141104" Jul 24 15:54:34.391705 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.391699 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="953175da-3c79-4f99-a3b2-094491ec0cb6" containerName="test-curl-7966-140546768141104" Jul 24 15:54:34.394722 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.394704 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:34.399953 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.399924 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:34.407644 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.407612 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwq9c\" (UniqueName: \"kubernetes.io/projected/953175da-3c79-4f99-a3b2-094491ec0cb6-kube-api-access-bwq9c\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:54:34.437061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.437023 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953175da-3c79-4f99-a3b2-094491ec0cb6" path="/var/lib/kubelet/pods/953175da-3c79-4f99-a3b2-094491ec0cb6/volumes" Jul 24 15:54:34.486588 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:54:34.486547 2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod953175da_3c79_4f99_a3b2_094491ec0cb6.slice\": RecentStats: unable to find data in memory cache]" Jul 24 15:54:34.508057 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.508022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq52g\" (UniqueName: \"kubernetes.io/projected/22fdfa09-3e72-46ab-9bb0-479062647da7-kube-api-access-bq52g\") pod \"test-curl-7966-140546774148848\" (UID: \"22fdfa09-3e72-46ab-9bb0-479062647da7\") " pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:34.608765 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.608670 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq52g\" (UniqueName: \"kubernetes.io/projected/22fdfa09-3e72-46ab-9bb0-479062647da7-kube-api-access-bq52g\") pod \"test-curl-7966-140546774148848\" (UID: \"22fdfa09-3e72-46ab-9bb0-479062647da7\") " pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:34.616780 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.616748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq52g\" (UniqueName: \"kubernetes.io/projected/22fdfa09-3e72-46ab-9bb0-479062647da7-kube-api-access-bq52g\") pod \"test-curl-7966-140546774148848\" (UID: \"22fdfa09-3e72-46ab-9bb0-479062647da7\") " pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:34.707437 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.707394 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:34.849972 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:34.849942 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:34.851709 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:34.851680 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22fdfa09_3e72_46ab_9bb0_479062647da7.slice/crio-f84bb97866ffc1b34003c7bb457e9dfffaf3dfc310660e149ce3b2a933704226 WatchSource:0}: Error finding container f84bb97866ffc1b34003c7bb457e9dfffaf3dfc310660e149ce3b2a933704226: Status 404 returned error can't find the container with id f84bb97866ffc1b34003c7bb457e9dfffaf3dfc310660e149ce3b2a933704226 Jul 24 15:54:35.058318 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:35.058288 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768141104" Jul 24 15:54:35.058318 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:35.058317 2563 scope.go:117] "RemoveContainer" containerID="1902b69ba12a3a3314d654d85252db930aa9775f300be3b92f9c06ce8331f9d7" Jul 24 15:54:35.059846 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:35.059822 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546774148848" event={"ID":"22fdfa09-3e72-46ab-9bb0-479062647da7","Type":"ContainerStarted","Data":"f84bb97866ffc1b34003c7bb457e9dfffaf3dfc310660e149ce3b2a933704226"} Jul 24 15:54:36.067868 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:36.067829 2563 generic.go:356] "Generic (PLEG): container finished" podID="22fdfa09-3e72-46ab-9bb0-479062647da7" containerID="5a563d4a946b025766400e508032880d33ff05c5172621bf40179cd084e19e20" exitCode=0 Jul 24 15:54:36.068355 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:36.067913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546774148848" event={"ID":"22fdfa09-3e72-46ab-9bb0-479062647da7","Type":"ContainerDied","Data":"5a563d4a946b025766400e508032880d33ff05c5172621bf40179cd084e19e20"} Jul 24 15:54:37.220652 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.220623 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:37.242301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.242272 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7966-140546774148848_22fdfa09-3e72-46ab-9bb0-479062647da7/test-curl-7966-140546774148848/0.log" Jul 24 15:54:37.264346 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.264307 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:37.267730 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.267695 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:37.335009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.334908 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq52g\" (UniqueName: \"kubernetes.io/projected/22fdfa09-3e72-46ab-9bb0-479062647da7-kube-api-access-bq52g\") pod \"22fdfa09-3e72-46ab-9bb0-479062647da7\" (UID: \"22fdfa09-3e72-46ab-9bb0-479062647da7\") " Jul 24 15:54:37.337004 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.336969 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fdfa09-3e72-46ab-9bb0-479062647da7-kube-api-access-bq52g" (OuterVolumeSpecName: "kube-api-access-bq52g") pod "22fdfa09-3e72-46ab-9bb0-479062647da7" (UID: "22fdfa09-3e72-46ab-9bb0-479062647da7"). InnerVolumeSpecName "kube-api-access-bq52g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:37.379601 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.379557 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:37.380039 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.380024 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fdfa09-3e72-46ab-9bb0-479062647da7" containerName="test-curl-7966-140546774148848" Jul 24 15:54:37.380086 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.380040 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fdfa09-3e72-46ab-9bb0-479062647da7" containerName="test-curl-7966-140546774148848" Jul 24 15:54:37.380118 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.380113 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fdfa09-3e72-46ab-9bb0-479062647da7" containerName="test-curl-7966-140546774148848" Jul 24 15:54:37.383212 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.383193 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:37.391265 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.391214 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:37.436302 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.436230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzj2l\" (UniqueName: \"kubernetes.io/projected/47a8395a-c5c6-475c-870e-c2e7ed3a8d03-kube-api-access-vzj2l\") pod \"test-curl-7966-140546774148848\" (UID: \"47a8395a-c5c6-475c-870e-c2e7ed3a8d03\") " pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:37.436488 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.436350 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq52g\" (UniqueName: \"kubernetes.io/projected/22fdfa09-3e72-46ab-9bb0-479062647da7-kube-api-access-bq52g\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:54:37.537093 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.537048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzj2l\" (UniqueName: \"kubernetes.io/projected/47a8395a-c5c6-475c-870e-c2e7ed3a8d03-kube-api-access-vzj2l\") pod \"test-curl-7966-140546774148848\" (UID: \"47a8395a-c5c6-475c-870e-c2e7ed3a8d03\") " pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:37.553026 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.552990 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzj2l\" (UniqueName: \"kubernetes.io/projected/47a8395a-c5c6-475c-870e-c2e7ed3a8d03-kube-api-access-vzj2l\") pod \"test-curl-7966-140546774148848\" (UID: \"47a8395a-c5c6-475c-870e-c2e7ed3a8d03\") " pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:37.694385 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.694227 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:37.837192 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:37.837166 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:37.839281 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:37.839238 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47a8395a_c5c6_475c_870e_c2e7ed3a8d03.slice/crio-b1478cd80e871610cca89455dfe282fc794414d104f7c53823f4d62851985475 WatchSource:0}: Error finding container b1478cd80e871610cca89455dfe282fc794414d104f7c53823f4d62851985475: Status 404 returned error can't find the container with id b1478cd80e871610cca89455dfe282fc794414d104f7c53823f4d62851985475 Jul 24 15:54:38.085319 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:38.085279 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546774148848" event={"ID":"47a8395a-c5c6-475c-870e-c2e7ed3a8d03","Type":"ContainerStarted","Data":"b1478cd80e871610cca89455dfe282fc794414d104f7c53823f4d62851985475"} Jul 24 15:54:38.086530 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:38.086500 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:38.086530 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:38.086517 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f84bb97866ffc1b34003c7bb457e9dfffaf3dfc310660e149ce3b2a933704226" Jul 24 15:54:38.098534 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:38.098500 2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7966-140546774148848" oldPodUID="22fdfa09-3e72-46ab-9bb0-479062647da7" podUID="47a8395a-c5c6-475c-870e-c2e7ed3a8d03" Jul 24 15:54:38.437956 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:38.437868 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22fdfa09-3e72-46ab-9bb0-479062647da7" path="/var/lib/kubelet/pods/22fdfa09-3e72-46ab-9bb0-479062647da7/volumes" Jul 24 15:54:38.764310 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:38.764269 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:39.093657 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:39.093559 2563 generic.go:356] "Generic (PLEG): container finished" podID="47a8395a-c5c6-475c-870e-c2e7ed3a8d03" containerID="6a38fd1a981b424609ee9d037bc9790a2593170d7ad71fe5c55088f559445392" exitCode=0 Jul 24 15:54:39.093657 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:39.093643 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546774148848" event={"ID":"47a8395a-c5c6-475c-870e-c2e7ed3a8d03","Type":"ContainerDied","Data":"6a38fd1a981b424609ee9d037bc9790a2593170d7ad71fe5c55088f559445392"} Jul 24 15:54:40.241147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.241117 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:40.261164 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.261131 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7966-140546774148848_47a8395a-c5c6-475c-870e-c2e7ed3a8d03/test-curl-7966-140546774148848/0.log" Jul 24 15:54:40.287316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.287273 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:40.290887 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.290860 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7966-140546774148848"] Jul 24 15:54:40.361511 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.361403 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzj2l\" (UniqueName: \"kubernetes.io/projected/47a8395a-c5c6-475c-870e-c2e7ed3a8d03-kube-api-access-vzj2l\") pod \"47a8395a-c5c6-475c-870e-c2e7ed3a8d03\" (UID: \"47a8395a-c5c6-475c-870e-c2e7ed3a8d03\") " Jul 24 15:54:40.363540 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.363511 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a8395a-c5c6-475c-870e-c2e7ed3a8d03-kube-api-access-vzj2l" (OuterVolumeSpecName: "kube-api-access-vzj2l") pod "47a8395a-c5c6-475c-870e-c2e7ed3a8d03" (UID: "47a8395a-c5c6-475c-870e-c2e7ed3a8d03"). InnerVolumeSpecName "kube-api-access-vzj2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:40.411802 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.411767 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:40.412274 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.412236 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a8395a-c5c6-475c-870e-c2e7ed3a8d03" containerName="test-curl-7966-140546774148848" Jul 24 15:54:40.412328 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.412276 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a8395a-c5c6-475c-870e-c2e7ed3a8d03" containerName="test-curl-7966-140546774148848" Jul 24 15:54:40.412388 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.412379 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a8395a-c5c6-475c-870e-c2e7ed3a8d03" containerName="test-curl-7966-140546774148848" Jul 24 15:54:40.415429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.415405 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:40.423892 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.423866 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:40.438736 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.438692 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a8395a-c5c6-475c-870e-c2e7ed3a8d03" path="/var/lib/kubelet/pods/47a8395a-c5c6-475c-870e-c2e7ed3a8d03/volumes" Jul 24 15:54:40.462181 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.462149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86xm\" (UniqueName: \"kubernetes.io/projected/3ca6d5ac-c284-4b08-88c4-f945379363f7-kube-api-access-n86xm\") pod \"test-curl-7966-140546768123344\" (UID: \"3ca6d5ac-c284-4b08-88c4-f945379363f7\") " pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:40.462367 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.462211 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzj2l\" (UniqueName: \"kubernetes.io/projected/47a8395a-c5c6-475c-870e-c2e7ed3a8d03-kube-api-access-vzj2l\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:54:40.563573 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.563529 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86xm\" (UniqueName: \"kubernetes.io/projected/3ca6d5ac-c284-4b08-88c4-f945379363f7-kube-api-access-n86xm\") pod \"test-curl-7966-140546768123344\" (UID: \"3ca6d5ac-c284-4b08-88c4-f945379363f7\") " pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:40.571849 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.571820 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86xm\" (UniqueName: \"kubernetes.io/projected/3ca6d5ac-c284-4b08-88c4-f945379363f7-kube-api-access-n86xm\") pod \"test-curl-7966-140546768123344\" (UID: \"3ca6d5ac-c284-4b08-88c4-f945379363f7\") " pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:40.727163 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.727071 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:40.869012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:40.868966 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:40.874865 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:40.874829 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca6d5ac_c284_4b08_88c4_f945379363f7.slice/crio-def2d17ade4d6b87524a6a27a2fc1cc0ab46a12a917dc73ab9230d7243331a39 WatchSource:0}: Error finding container def2d17ade4d6b87524a6a27a2fc1cc0ab46a12a917dc73ab9230d7243331a39: Status 404 returned error can't find the container with id def2d17ade4d6b87524a6a27a2fc1cc0ab46a12a917dc73ab9230d7243331a39 Jul 24 15:54:41.105877 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:41.105843 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546774148848" Jul 24 15:54:41.106056 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:41.105891 2563 scope.go:117] "RemoveContainer" containerID="6a38fd1a981b424609ee9d037bc9790a2593170d7ad71fe5c55088f559445392" Jul 24 15:54:41.107296 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:41.107254 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768123344" event={"ID":"3ca6d5ac-c284-4b08-88c4-f945379363f7","Type":"ContainerStarted","Data":"def2d17ade4d6b87524a6a27a2fc1cc0ab46a12a917dc73ab9230d7243331a39"} Jul 24 15:54:42.114351 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:42.114312 2563 generic.go:356] "Generic (PLEG): container finished" podID="3ca6d5ac-c284-4b08-88c4-f945379363f7" containerID="02749ebebbb2d9b2505f64bca930a3572a99878cfb5cc6f9a19b71d9e0392893" exitCode=0 Jul 24 15:54:42.114853 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:42.114398 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768123344" event={"ID":"3ca6d5ac-c284-4b08-88c4-f945379363f7","Type":"ContainerDied","Data":"02749ebebbb2d9b2505f64bca930a3572a99878cfb5cc6f9a19b71d9e0392893"} Jul 24 15:54:43.269735 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.269707 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:43.290731 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.290685 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7966-140546768123344_3ca6d5ac-c284-4b08-88c4-f945379363f7/test-curl-7966-140546768123344/0.log" Jul 24 15:54:43.317632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.317597 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:43.319954 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.319928 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:43.395582 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.395486 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n86xm\" (UniqueName: \"kubernetes.io/projected/3ca6d5ac-c284-4b08-88c4-f945379363f7-kube-api-access-n86xm\") pod \"3ca6d5ac-c284-4b08-88c4-f945379363f7\" (UID: \"3ca6d5ac-c284-4b08-88c4-f945379363f7\") " Jul 24 15:54:43.397612 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.397572 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca6d5ac-c284-4b08-88c4-f945379363f7-kube-api-access-n86xm" (OuterVolumeSpecName: "kube-api-access-n86xm") pod "3ca6d5ac-c284-4b08-88c4-f945379363f7" (UID: "3ca6d5ac-c284-4b08-88c4-f945379363f7"). InnerVolumeSpecName "kube-api-access-n86xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:43.433707 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.433670 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:43.434092 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.434078 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca6d5ac-c284-4b08-88c4-f945379363f7" containerName="test-curl-7966-140546768123344" Jul 24 15:54:43.434138 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.434094 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca6d5ac-c284-4b08-88c4-f945379363f7" containerName="test-curl-7966-140546768123344" Jul 24 15:54:43.434184 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.434175 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca6d5ac-c284-4b08-88c4-f945379363f7" containerName="test-curl-7966-140546768123344" Jul 24 15:54:43.437236 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.437209 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:43.444924 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.444896 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:43.497012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.496970 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4vmx\" (UniqueName: \"kubernetes.io/projected/9749ade9-7770-4d11-a59a-cd4d4450ee70-kube-api-access-z4vmx\") pod \"test-curl-7966-140546768123344\" (UID: \"9749ade9-7770-4d11-a59a-cd4d4450ee70\") " pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:43.497173 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.497119 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86xm\" (UniqueName: \"kubernetes.io/projected/3ca6d5ac-c284-4b08-88c4-f945379363f7-kube-api-access-n86xm\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:54:43.597588 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.597549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4vmx\" (UniqueName: \"kubernetes.io/projected/9749ade9-7770-4d11-a59a-cd4d4450ee70-kube-api-access-z4vmx\") pod \"test-curl-7966-140546768123344\" (UID: \"9749ade9-7770-4d11-a59a-cd4d4450ee70\") " pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:43.605948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.605923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4vmx\" (UniqueName: \"kubernetes.io/projected/9749ade9-7770-4d11-a59a-cd4d4450ee70-kube-api-access-z4vmx\") pod \"test-curl-7966-140546768123344\" (UID: \"9749ade9-7770-4d11-a59a-cd4d4450ee70\") " pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:43.748229 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.748132 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:43.890547 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:43.890517 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:43.892021 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:43.891991 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9749ade9_7770_4d11_a59a_cd4d4450ee70.slice/crio-74107b3637e945b4ff32fdd937645176efe65f65183677c81c24ff6fe75421f1 WatchSource:0}: Error finding container 74107b3637e945b4ff32fdd937645176efe65f65183677c81c24ff6fe75421f1: Status 404 returned error can't find the container with id 74107b3637e945b4ff32fdd937645176efe65f65183677c81c24ff6fe75421f1 Jul 24 15:54:44.131199 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:44.131166 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:44.131392 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:44.131207 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="def2d17ade4d6b87524a6a27a2fc1cc0ab46a12a917dc73ab9230d7243331a39" Jul 24 15:54:44.132475 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:44.132443 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768123344" event={"ID":"9749ade9-7770-4d11-a59a-cd4d4450ee70","Type":"ContainerStarted","Data":"74107b3637e945b4ff32fdd937645176efe65f65183677c81c24ff6fe75421f1"} Jul 24 15:54:44.143948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:44.143918 2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7966-140546768123344" oldPodUID="3ca6d5ac-c284-4b08-88c4-f945379363f7" podUID="9749ade9-7770-4d11-a59a-cd4d4450ee70" Jul 24 15:54:44.437630 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:44.437543 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca6d5ac-c284-4b08-88c4-f945379363f7" path="/var/lib/kubelet/pods/3ca6d5ac-c284-4b08-88c4-f945379363f7/volumes" Jul 24 15:54:45.145057 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:45.145008 2563 generic.go:356] "Generic (PLEG): container finished" podID="9749ade9-7770-4d11-a59a-cd4d4450ee70" containerID="cc2893fbeda780bdf2fc621a493cca128a86c412c3c056aa10ffdf63438f2292" exitCode=0 Jul 24 15:54:45.145386 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:45.145358 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7966-140546768123344" event={"ID":"9749ade9-7770-4d11-a59a-cd4d4450ee70","Type":"ContainerDied","Data":"cc2893fbeda780bdf2fc621a493cca128a86c412c3c056aa10ffdf63438f2292"} Jul 24 15:54:46.298233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.298205 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:46.317920 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.317884 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7966-140546768123344_9749ade9-7770-4d11-a59a-cd4d4450ee70/test-curl-7966-140546768123344/0.log" Jul 24 15:54:46.340097 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.340049 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:46.343718 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.343685 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7966-140546768123344"] Jul 24 15:54:46.421880 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.421779 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4vmx\" (UniqueName: \"kubernetes.io/projected/9749ade9-7770-4d11-a59a-cd4d4450ee70-kube-api-access-z4vmx\") pod \"9749ade9-7770-4d11-a59a-cd4d4450ee70\" (UID: \"9749ade9-7770-4d11-a59a-cd4d4450ee70\") " Jul 24 15:54:46.424036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.424005 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9749ade9-7770-4d11-a59a-cd4d4450ee70-kube-api-access-z4vmx" (OuterVolumeSpecName: "kube-api-access-z4vmx") pod "9749ade9-7770-4d11-a59a-cd4d4450ee70" (UID: "9749ade9-7770-4d11-a59a-cd4d4450ee70"). InnerVolumeSpecName "kube-api-access-z4vmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:46.438819 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.438780 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9749ade9-7770-4d11-a59a-cd4d4450ee70" path="/var/lib/kubelet/pods/9749ade9-7770-4d11-a59a-cd4d4450ee70/volumes" Jul 24 15:54:46.522833 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.522791 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4vmx\" (UniqueName: \"kubernetes.io/projected/9749ade9-7770-4d11-a59a-cd4d4450ee70-kube-api-access-z4vmx\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:54:46.842722 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:46.842677 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:47.158734 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:47.158641 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7966-140546768123344" Jul 24 15:54:47.158734 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:47.158652 2563 scope.go:117] "RemoveContainer" containerID="cc2893fbeda780bdf2fc621a493cca128a86c412c3c056aa10ffdf63438f2292" Jul 24 15:54:47.172497 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:47.172462 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:47.530946 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:47.530909 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:49.890683 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:49.890648 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:49.984084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:49.984053 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:50.200321 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.200207 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts"] Jul 24 15:54:50.200702 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.200686 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9749ade9-7770-4d11-a59a-cd4d4450ee70" containerName="test-curl-7966-140546768123344" Jul 24 15:54:50.200745 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.200705 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9749ade9-7770-4d11-a59a-cd4d4450ee70" containerName="test-curl-7966-140546768123344" Jul 24 15:54:50.200789 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.200779 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9749ade9-7770-4d11-a59a-cd4d4450ee70" containerName="test-curl-7966-140546768123344" Jul 24 15:54:50.205132 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.205109 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.215361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.215337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a64ba4-serving-cert\"" Jul 24 15:54:50.223560 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.223527 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts"] Jul 24 15:54:50.250565 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.250515 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:50.360538 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.360499 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55k7c\" (UniqueName: \"kubernetes.io/projected/f2cb84eb-4da3-42b4-9982-93e9fe116dba-kube-api-access-55k7c\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.360538 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.360543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.360817 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.360666 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb84eb-4da3-42b4-9982-93e9fe116dba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.461578 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.461481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55k7c\" (UniqueName: \"kubernetes.io/projected/f2cb84eb-4da3-42b4-9982-93e9fe116dba-kube-api-access-55k7c\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.461578 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.461529 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.461790 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.461597 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb84eb-4da3-42b4-9982-93e9fe116dba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.461790 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:54:50.461674 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-serving-cert: secret "maas-api-e2e-ipp-a-a64ba4-serving-cert" not found Jul 24 15:54:50.461790 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:54:50.461756 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls podName:f2cb84eb-4da3-42b4-9982-93e9fe116dba nodeName:}" failed. No retries permitted until 2026-07-24 15:54:50.961737183 +0000 UTC m=+2805.048389076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls") pod "maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" (UID: "f2cb84eb-4da3-42b4-9982-93e9fe116dba") : secret "maas-api-e2e-ipp-a-a64ba4-serving-cert" not found Jul 24 15:54:50.462176 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.462157 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb84eb-4da3-42b4-9982-93e9fe116dba-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.477945 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.477912 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55k7c\" (UniqueName: \"kubernetes.io/projected/f2cb84eb-4da3-42b4-9982-93e9fe116dba-kube-api-access-55k7c\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.570708 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.570664 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:50.891918 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.891877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:50.966440 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.966403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.968961 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.968924 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:50.981008 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.980972 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:50.982684 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.982661 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx"] Jul 24 15:54:50.986536 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.986518 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:50.988873 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.988853 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a64ba4\"" Jul 24 15:54:50.997477 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:50.997445 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx"] Jul 24 15:54:51.067076 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.067034 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzln\" (UniqueName: \"kubernetes.io/projected/ade04c84-a912-46a5-a7c7-87c9f3b2838d-kube-api-access-2dzln\") pod \"payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.067272 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.067104 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade04c84-a912-46a5-a7c7-87c9f3b2838d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.116314 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.116263 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:51.168298 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.168199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzln\" (UniqueName: \"kubernetes.io/projected/ade04c84-a912-46a5-a7c7-87c9f3b2838d-kube-api-access-2dzln\") pod \"payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.168468 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.168349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade04c84-a912-46a5-a7c7-87c9f3b2838d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.169347 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.169316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade04c84-a912-46a5-a7c7-87c9f3b2838d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.177212 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.177178 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzln\" (UniqueName: \"kubernetes.io/projected/ade04c84-a912-46a5-a7c7-87c9f3b2838d-kube-api-access-2dzln\") pod \"payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.280436 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.280408 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts"] Jul 24 15:54:51.283104 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:51.283073 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cb84eb_4da3_42b4_9982_93e9fe116dba.slice/crio-02864834423e0376ccb029c373db421c1b2b6ea308f2a9d7607d9303f6588e41 WatchSource:0}: Error finding container 02864834423e0376ccb029c373db421c1b2b6ea308f2a9d7607d9303f6588e41: Status 404 returned error can't find the container with id 02864834423e0376ccb029c373db421c1b2b6ea308f2a9d7607d9303f6588e41 Jul 24 15:54:51.299320 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.299288 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:51.448646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:51.448613 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx"] Jul 24 15:54:51.450113 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:54:51.450078 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade04c84_a912_46a5_a7c7_87c9f3b2838d.slice/crio-ce50677b146f351b50166d03ba8d298ebec0da7ab5d095a6222ce819f54ab1ba WatchSource:0}: Error finding container ce50677b146f351b50166d03ba8d298ebec0da7ab5d095a6222ce819f54ab1ba: Status 404 returned error can't find the container with id ce50677b146f351b50166d03ba8d298ebec0da7ab5d095a6222ce819f54ab1ba Jul 24 15:54:52.202944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.202890 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" event={"ID":"f2cb84eb-4da3-42b4-9982-93e9fe116dba","Type":"ContainerStarted","Data":"0d5bca2692389392e78fa5b4a875ca03813663f325a1b5b1b13fb6aebb98296b"} Jul 24 15:54:52.203445 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.202960 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" event={"ID":"f2cb84eb-4da3-42b4-9982-93e9fe116dba","Type":"ContainerStarted","Data":"02864834423e0376ccb029c373db421c1b2b6ea308f2a9d7607d9303f6588e41"} Jul 24 15:54:52.203445 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.203088 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:54:52.204627 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.204600 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" event={"ID":"ade04c84-a912-46a5-a7c7-87c9f3b2838d","Type":"ContainerStarted","Data":"2b6ce461ab4536d3274933f592c1fa2808de190b4aada9e672749cd72ca67655"} Jul 24 15:54:52.204735 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.204650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" event={"ID":"ade04c84-a912-46a5-a7c7-87c9f3b2838d","Type":"ContainerStarted","Data":"ce50677b146f351b50166d03ba8d298ebec0da7ab5d095a6222ce819f54ab1ba"} Jul 24 15:54:52.204735 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.204681 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:54:52.223796 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.223739 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" podStartSLOduration=1.768426673 podStartE2EDuration="2.223722637s" totalImagesPullingTime="455.295964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:52.220704238 +0000 UTC m=+2806.307356151" watchObservedRunningTime="2026-07-24 15:54:52.223722637 +0000 UTC m=+2806.310374551" Jul 24 15:54:52.236518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:52.236466 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" podStartSLOduration=2.236446411 podStartE2EDuration="2.236446411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:52.236110354 +0000 UTC m=+2806.322762269" watchObservedRunningTime="2026-07-24 15:54:52.236446411 +0000 UTC m=+2806.323098324" Jul 24 15:54:54.379109 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:54.379076 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:54:58.217233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:54:58.217199 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:55:03.212942 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:03.212909 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:55:05.286830 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:05.286792 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:05.613209 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:05.613098 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:05.807632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:05.807597 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:09.052779 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.052710 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:09.217430 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.217395 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:09.255037 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.255003 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7"] Jul 24 15:55:09.261104 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.261079 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.263961 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.263922 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4dda13-serving-cert\"" Jul 24 15:55:09.270523 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.270494 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7"] Jul 24 15:55:09.318130 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.318035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:09.339082 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.339036 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.339309 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.339107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rptc\" (UniqueName: \"kubernetes.io/projected/0b564b77-c368-4b46-a097-bda5c17496fb-kube-api-access-9rptc\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.339309 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.339141 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b564b77-c368-4b46-a097-bda5c17496fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.439646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.439605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rptc\" (UniqueName: \"kubernetes.io/projected/0b564b77-c368-4b46-a097-bda5c17496fb-kube-api-access-9rptc\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.439865 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.439654 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b564b77-c368-4b46-a097-bda5c17496fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.439865 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.439812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.440002 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:55:09.439910 2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-serving-cert: secret "maas-api-e2e-ipp-b-4dda13-serving-cert" not found Jul 24 15:55:09.440002 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:55:09.439972 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls podName:0b564b77-c368-4b46-a097-bda5c17496fb nodeName:}" failed. No retries permitted until 2026-07-24 15:55:09.939953315 +0000 UTC m=+2824.026605208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls") pod "maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" (UID: "0b564b77-c368-4b46-a097-bda5c17496fb") : secret "maas-api-e2e-ipp-b-4dda13-serving-cert" not found Jul 24 15:55:09.440412 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.440392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b564b77-c368-4b46-a097-bda5c17496fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.449707 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.449667 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rptc\" (UniqueName: \"kubernetes.io/projected/0b564b77-c368-4b46-a097-bda5c17496fb-kube-api-access-9rptc\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.784084 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.784035 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:09.888128 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.888090 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:09.943932 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.943878 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.946456 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.946425 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:09.964545 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:09.964509 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:10.174183 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:10.174073 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:10.328646 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:10.328612 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7"] Jul 24 15:55:10.329817 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:55:10.329785 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b564b77_c368_4b46_a097_bda5c17496fb.slice/crio-7ab32548dcfb6b93cdc6163f6b5f04e14e54bfce0013f369a66fd72a330f4e5c WatchSource:0}: Error finding container 7ab32548dcfb6b93cdc6163f6b5f04e14e54bfce0013f369a66fd72a330f4e5c: Status 404 returned error can't find the container with id 7ab32548dcfb6b93cdc6163f6b5f04e14e54bfce0013f369a66fd72a330f4e5c Jul 24 15:55:11.323827 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:11.323781 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" event={"ID":"0b564b77-c368-4b46-a097-bda5c17496fb","Type":"ContainerStarted","Data":"c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21"} Jul 24 15:55:11.324279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:11.323850 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:11.324279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:11.323870 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" event={"ID":"0b564b77-c368-4b46-a097-bda5c17496fb","Type":"ContainerStarted","Data":"7ab32548dcfb6b93cdc6163f6b5f04e14e54bfce0013f369a66fd72a330f4e5c"} Jul 24 15:55:11.341464 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:11.341400 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" podStartSLOduration=1.843696089 podStartE2EDuration="2.341382194s" totalImagesPullingTime="497.686105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:11.340142826 +0000 UTC m=+2825.426794738" watchObservedRunningTime="2026-07-24 15:55:11.341382194 +0000 UTC m=+2825.428034108" Jul 24 15:55:13.506483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:13.506451 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:55:17.340983 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:17.340948 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:55:30.622116 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:30.622085 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hl4mb_22baeaaa-964a-44be-bc00-89064290f48c/payload-processing/0.log" Jul 24 15:55:53.050764 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:55:53.050727 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:03.802483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:03.802396 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:32.938186 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:32.938142 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hl4mb_22baeaaa-964a-44be-bc00-89064290f48c/payload-processing/0.log" Jul 24 15:56:33.323729 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.323693 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:33.604264 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.604129 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66"] Jul 24 15:56:33.607937 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.607915 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.610196 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.610170 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-6b1d8d-openshift-default-dockercfg-lp752\"" Jul 24 15:56:33.619644 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.619612 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66"] Jul 24 15:56:33.640872 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.640834 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:33.729824 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729788 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-credential-socket\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729870 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-envoy\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729890 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-podinfo\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729933 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-data\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f672a9-5a20-4d10-b5ea-95b676127c59-istiod-ca-cert\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pht2w\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-kube-api-access-pht2w\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.729990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-socket\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730224 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.730049 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-certs\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.730224 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.730065 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-token\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.821362 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.821325 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:33.831115 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-certs\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831115 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-token\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-credential-socket\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-envoy\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831209 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-podinfo\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831281 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-data\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831297 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f672a9-5a20-4d10-b5ea-95b676127c59-istiod-ca-cert\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pht2w\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-kube-api-access-pht2w\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-socket\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831730 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831535 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-certs\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831730 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831597 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-credential-socket\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831723 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-socket\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.831803 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.831750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-data\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.832130 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.832109 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f672a9-5a20-4d10-b5ea-95b676127c59-istiod-ca-cert\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.833633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.833615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-envoy\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.833893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.833874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-podinfo\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.839424 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.839394 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-token\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.839716 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.839690 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pht2w\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-kube-api-access-pht2w\") pod \"e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:33.920281 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:33.920151 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:34.068443 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.068413 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66"] Jul 24 15:56:34.069896 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:56:34.069870 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f672a9_5a20_4d10_b5ea_95b676127c59.slice/crio-b7681d920e98c5ab7a0d84c2673e1ee878bfda0bf1b41e3fd787d5a76b3ea441 WatchSource:0}: Error finding container b7681d920e98c5ab7a0d84c2673e1ee878bfda0bf1b41e3fd787d5a76b3ea441: Status 404 returned error can't find the container with id b7681d920e98c5ab7a0d84c2673e1ee878bfda0bf1b41e3fd787d5a76b3ea441 Jul 24 15:56:34.072275 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.072227 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:56:34.072338 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.072313 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:56:34.072382 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.072338 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:56:34.835780 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.835742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" event={"ID":"68f672a9-5a20-4d10-b5ea-95b676127c59","Type":"ContainerStarted","Data":"44239dc337524c68dc5b69f4497172b4239b1c86573f3b25430496c21de213aa"} Jul 24 15:56:34.836054 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.835798 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" event={"ID":"68f672a9-5a20-4d10-b5ea-95b676127c59","Type":"ContainerStarted","Data":"b7681d920e98c5ab7a0d84c2673e1ee878bfda0bf1b41e3fd787d5a76b3ea441"} Jul 24 15:56:34.858157 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.858097 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" podStartSLOduration=1.8580784449999999 podStartE2EDuration="1.858078445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:34.856019382 +0000 UTC m=+2908.942671297" watchObservedRunningTime="2026-07-24 15:56:34.858078445 +0000 UTC m=+2908.944730359" Jul 24 15:56:34.920833 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:34.920790 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:35.926204 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:35.926172 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:36.767727 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.767683 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:36.851399 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.851365 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:36.852539 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.852498 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:36.853782 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.853752 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:36.857074 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.857045 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn"] Jul 24 15:56:36.861639 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.861616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:36.863793 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.863769 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-6b1d8d-serving-cert\"" Jul 24 15:56:36.869222 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.869191 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn"] Jul 24 15:56:36.930871 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.930842 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:36.960455 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.960405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:36.960669 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.960644 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfrj\" (UniqueName: \"kubernetes.io/projected/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-kube-api-access-7pfrj\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:36.960714 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:36.960687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.031012 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.030921 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:37.061543 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.061504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.061749 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.061635 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfrj\" (UniqueName: \"kubernetes.io/projected/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-kube-api-access-7pfrj\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.061749 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.061669 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.062305 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.062277 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.064496 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.064472 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.074081 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.074051 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfrj\" (UniqueName: \"kubernetes.io/projected/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-kube-api-access-7pfrj\") pod \"maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.174942 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.174899 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:37.334100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.334069 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn"] Jul 24 15:56:37.335972 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:56:37.335938 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79dbca9_6723_489d_bfb4_ded15fdcbf6c.slice/crio-3440bc16f6a1930db591d796c0b77b11d56c728d4f471e61d3d543e8fab1481f WatchSource:0}: Error finding container 3440bc16f6a1930db591d796c0b77b11d56c728d4f471e61d3d543e8fab1481f: Status 404 returned error can't find the container with id 3440bc16f6a1930db591d796c0b77b11d56c728d4f471e61d3d543e8fab1481f Jul 24 15:56:37.466159 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.466120 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:37.554682 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.554645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:37.556718 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.556686 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg"] Jul 24 15:56:37.561672 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.561640 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.564149 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.564126 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-6b1d8d\"" Jul 24 15:56:37.564316 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.564190 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-6b1d8d-dockercfg-t69nd\"" Jul 24 15:56:37.568489 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.568465 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg"] Jul 24 15:56:37.668879 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.668775 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9d8c\" (UniqueName: \"kubernetes.io/projected/e1c791db-783c-46c5-93a3-2e1c6d76879a-kube-api-access-q9d8c\") pod \"payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.668879 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.668824 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c791db-783c-46c5-93a3-2e1c6d76879a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.769486 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.769443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9d8c\" (UniqueName: \"kubernetes.io/projected/e1c791db-783c-46c5-93a3-2e1c6d76879a-kube-api-access-q9d8c\") pod \"payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.769688 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.769502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c791db-783c-46c5-93a3-2e1c6d76879a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.770303 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.770279 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c791db-783c-46c5-93a3-2e1c6d76879a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.778396 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.778365 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9d8c\" (UniqueName: \"kubernetes.io/projected/e1c791db-783c-46c5-93a3-2e1c6d76879a-kube-api-access-q9d8c\") pod \"payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:37.859751 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.859705 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" event={"ID":"a79dbca9-6723-489d-bfb4-ded15fdcbf6c","Type":"ContainerStarted","Data":"3440bc16f6a1930db591d796c0b77b11d56c728d4f471e61d3d543e8fab1481f"} Jul 24 15:56:37.875449 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:37.875410 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:38.030188 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.030157 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg"] Jul 24 15:56:38.030883 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:56:38.030855 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c791db_783c_46c5_93a3_2e1c6d76879a.slice/crio-11d47c2912d9a6f97c40b19cc9f3af7a9d3571ab058236392078ef8fce83aaa1 WatchSource:0}: Error finding container 11d47c2912d9a6f97c40b19cc9f3af7a9d3571ab058236392078ef8fce83aaa1: Status 404 returned error can't find the container with id 11d47c2912d9a6f97c40b19cc9f3af7a9d3571ab058236392078ef8fce83aaa1 Jul 24 15:56:38.872119 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.872071 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" event={"ID":"a79dbca9-6723-489d-bfb4-ded15fdcbf6c","Type":"ContainerStarted","Data":"54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2"} Jul 24 15:56:38.872354 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.872341 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:38.874093 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.874056 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" event={"ID":"e1c791db-783c-46c5-93a3-2e1c6d76879a","Type":"ContainerStarted","Data":"82d52792758bedd6aa06339159b32242b9327fa2544e76e8a5c4a6fc873303a7"} Jul 24 15:56:38.874233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.874112 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" event={"ID":"e1c791db-783c-46c5-93a3-2e1c6d76879a","Type":"ContainerStarted","Data":"11d47c2912d9a6f97c40b19cc9f3af7a9d3571ab058236392078ef8fce83aaa1"} Jul 24 15:56:38.874233 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.874140 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:38.895381 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.895319 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" podStartSLOduration=2.420984603 podStartE2EDuration="2.895301483s" totalImagesPullingTime="474.31688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:38.893055239 +0000 UTC m=+2912.979707156" watchObservedRunningTime="2026-07-24 15:56:38.895301483 +0000 UTC m=+2912.981953396" Jul 24 15:56:38.909922 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:38.909864 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" podStartSLOduration=1.909845445 podStartE2EDuration="1.909845445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:38.908647829 +0000 UTC m=+2912.995299745" watchObservedRunningTime="2026-07-24 15:56:38.909845445 +0000 UTC m=+2912.996497356" Jul 24 15:56:41.247596 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:41.247563 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:56:44.887983 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:44.887949 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:56:49.883009 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.882974 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:56:49.947417 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.947373 2563 generic.go:356] "Generic (PLEG): container finished" podID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerID="44239dc337524c68dc5b69f4497172b4239b1c86573f3b25430496c21de213aa" exitCode=0 Jul 24 15:56:49.947604 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.947447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" event={"ID":"68f672a9-5a20-4d10-b5ea-95b676127c59","Type":"ContainerDied","Data":"44239dc337524c68dc5b69f4497172b4239b1c86573f3b25430496c21de213aa"} Jul 24 15:56:49.947877 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.947862 2563 scope.go:117] "RemoveContainer" containerID="44239dc337524c68dc5b69f4497172b4239b1c86573f3b25430496c21de213aa" Jul 24 15:56:49.949279 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.949234 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:56:49.949361 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.949334 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:56:49.949422 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:49.949405 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 24 15:56:50.955607 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:50.955575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" event={"ID":"68f672a9-5a20-4d10-b5ea-95b676127c59","Type":"ContainerStarted","Data":"ba6c3bd09ab1a854909e93134ca7ed035cb26e4ef5b0208288c5f18634f91ac4"} Jul 24 15:56:51.852121 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:51.852082 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:51.921305 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:51.921265 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:51.922687 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:51.922654 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 24 15:56:51.922847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:51.922715 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 24 15:56:52.921108 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:52.921067 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 24 15:56:52.921586 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:52.921126 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 24 15:56:53.920696 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:53.920648 2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 24 15:56:53.920928 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:53.920716 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 24 15:56:54.925579 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:54.925546 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:56:54.926628 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:56:54.926609 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:57:02.696780 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.696744 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn"] Jul 24 15:57:02.697257 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.697059 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" podUID="a79dbca9-6723-489d-bfb4-ded15fdcbf6c" containerName="maas-api" containerID="cri-o://54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2" gracePeriod=30 Jul 24 15:57:02.721876 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.721832 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:02.756107 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.756065 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg"] Jul 24 15:57:02.756397 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.756367 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" podUID="e1c791db-783c-46c5-93a3-2e1c6d76879a" containerName="payload-pre-processing" containerID="cri-o://82d52792758bedd6aa06339159b32242b9327fa2544e76e8a5c4a6fc873303a7" gracePeriod=30 Jul 24 15:57:02.822020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.821983 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:02.922881 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:02.922832 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:03.028236 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.028209 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:57:03.044159 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.044128 2563 generic.go:356] "Generic (PLEG): container finished" podID="e1c791db-783c-46c5-93a3-2e1c6d76879a" containerID="82d52792758bedd6aa06339159b32242b9327fa2544e76e8a5c4a6fc873303a7" exitCode=0 Jul 24 15:57:03.044363 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.044201 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" event={"ID":"e1c791db-783c-46c5-93a3-2e1c6d76879a","Type":"ContainerDied","Data":"82d52792758bedd6aa06339159b32242b9327fa2544e76e8a5c4a6fc873303a7"} Jul 24 15:57:03.053593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.053519 2563 generic.go:356] "Generic (PLEG): container finished" podID="a79dbca9-6723-489d-bfb4-ded15fdcbf6c" containerID="54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2" exitCode=0 Jul 24 15:57:03.053593 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.053583 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" Jul 24 15:57:03.053831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.053583 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" event={"ID":"a79dbca9-6723-489d-bfb4-ded15fdcbf6c","Type":"ContainerDied","Data":"54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2"} Jul 24 15:57:03.053831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.053727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn" event={"ID":"a79dbca9-6723-489d-bfb4-ded15fdcbf6c","Type":"ContainerDied","Data":"3440bc16f6a1930db591d796c0b77b11d56c728d4f471e61d3d543e8fab1481f"} Jul 24 15:57:03.053831 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.053758 2563 scope.go:117] "RemoveContainer" containerID="54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2" Jul 24 15:57:03.066673 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.066650 2563 scope.go:117] "RemoveContainer" containerID="54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2" Jul 24 15:57:03.067018 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:57:03.066996 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2\": container with ID starting with 54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2 not found: ID does not exist" containerID="54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2" Jul 24 15:57:03.067095 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.067034 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2"} err="failed to get container status \"54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2\": rpc error: code = NotFound desc = could not find container \"54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2\": container with ID starting with 54302928053f51b34d01a62ad46bcbae01113324e18339edc2e42003d63b7eb2 not found: ID does not exist" Jul 24 15:57:03.067815 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.067798 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:57:03.117852 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.117810 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9d8c\" (UniqueName: \"kubernetes.io/projected/e1c791db-783c-46c5-93a3-2e1c6d76879a-kube-api-access-q9d8c\") pod \"e1c791db-783c-46c5-93a3-2e1c6d76879a\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " Jul 24 15:57:03.118036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.117879 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c791db-783c-46c5-93a3-2e1c6d76879a-plugins-config-volume\") pod \"e1c791db-783c-46c5-93a3-2e1c6d76879a\" (UID: \"e1c791db-783c-46c5-93a3-2e1c6d76879a\") " Jul 24 15:57:03.118036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.117905 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-maas-api-tls\") pod \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " Jul 24 15:57:03.118036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.117935 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pfrj\" (UniqueName: \"kubernetes.io/projected/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-kube-api-access-7pfrj\") pod \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " Jul 24 15:57:03.118036 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.117953 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-openshift-service-ca\") pod \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\" (UID: \"a79dbca9-6723-489d-bfb4-ded15fdcbf6c\") " Jul 24 15:57:03.118364 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.118333 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c791db-783c-46c5-93a3-2e1c6d76879a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1c791db-783c-46c5-93a3-2e1c6d76879a" (UID: "e1c791db-783c-46c5-93a3-2e1c6d76879a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:57:03.118514 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.118430 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a79dbca9-6723-489d-bfb4-ded15fdcbf6c" (UID: "a79dbca9-6723-489d-bfb4-ded15fdcbf6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:57:03.120061 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.120031 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a79dbca9-6723-489d-bfb4-ded15fdcbf6c" (UID: "a79dbca9-6723-489d-bfb4-ded15fdcbf6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:57:03.120165 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.120061 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c791db-783c-46c5-93a3-2e1c6d76879a-kube-api-access-q9d8c" (OuterVolumeSpecName: "kube-api-access-q9d8c") pod "e1c791db-783c-46c5-93a3-2e1c6d76879a" (UID: "e1c791db-783c-46c5-93a3-2e1c6d76879a"). InnerVolumeSpecName "kube-api-access-q9d8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:57:03.120165 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.120087 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-kube-api-access-7pfrj" (OuterVolumeSpecName: "kube-api-access-7pfrj") pod "a79dbca9-6723-489d-bfb4-ded15fdcbf6c" (UID: "a79dbca9-6723-489d-bfb4-ded15fdcbf6c"). InnerVolumeSpecName "kube-api-access-7pfrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:57:03.219518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.219471 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9d8c\" (UniqueName: \"kubernetes.io/projected/e1c791db-783c-46c5-93a3-2e1c6d76879a-kube-api-access-q9d8c\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:03.219518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.219508 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c791db-783c-46c5-93a3-2e1c6d76879a-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:03.219518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.219518 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:03.219518 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.219529 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfrj\" (UniqueName: \"kubernetes.io/projected/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-kube-api-access-7pfrj\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:03.219807 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.219540 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a79dbca9-6723-489d-bfb4-ded15fdcbf6c-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:03.376365 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.376330 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn"] Jul 24 15:57:03.378904 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.378877 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b1d8d-669fd5745b-s62cn"] Jul 24 15:57:03.447318 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.447282 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:03.509877 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:03.509827 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:04.061485 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.061454 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" Jul 24 15:57:04.061944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.061483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg" event={"ID":"e1c791db-783c-46c5-93a3-2e1c6d76879a","Type":"ContainerDied","Data":"11d47c2912d9a6f97c40b19cc9f3af7a9d3571ab058236392078ef8fce83aaa1"} Jul 24 15:57:04.061944 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.061546 2563 scope.go:117] "RemoveContainer" containerID="82d52792758bedd6aa06339159b32242b9327fa2544e76e8a5c4a6fc873303a7" Jul 24 15:57:04.084063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.084026 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg"] Jul 24 15:57:04.087370 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.087335 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-6b1d8d-7b79666d8-m9hjg"] Jul 24 15:57:04.437736 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.437649 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79dbca9-6723-489d-bfb4-ded15fdcbf6c" path="/var/lib/kubelet/pods/a79dbca9-6723-489d-bfb4-ded15fdcbf6c/volumes" Jul 24 15:57:04.438020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:04.438006 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c791db-783c-46c5-93a3-2e1c6d76879a" path="/var/lib/kubelet/pods/e1c791db-783c-46c5-93a3-2e1c6d76879a/volumes" Jul 24 15:57:07.220501 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:07.220408 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:07.612695 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:07.612661 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:08.333774 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:08.333736 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:08.433847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:08.433817 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:10.940455 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:10.940420 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:13.663166 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:13.663133 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:13.906086 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:13.906048 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66"] Jul 24 15:57:13.906355 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:13.906320 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" containerID="cri-o://ba6c3bd09ab1a854909e93134ca7ed035cb26e4ef5b0208288c5f18634f91ac4" gracePeriod=30 Jul 24 15:57:15.100123 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:15.100091 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:19.155607 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.155574 2563 generic.go:356] "Generic (PLEG): container finished" podID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerID="ba6c3bd09ab1a854909e93134ca7ed035cb26e4ef5b0208288c5f18634f91ac4" exitCode=0 Jul 24 15:57:19.155994 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.155647 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" event={"ID":"68f672a9-5a20-4d10-b5ea-95b676127c59","Type":"ContainerDied","Data":"ba6c3bd09ab1a854909e93134ca7ed035cb26e4ef5b0208288c5f18634f91ac4"} Jul 24 15:57:19.155994 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.155701 2563 scope.go:117] "RemoveContainer" containerID="44239dc337524c68dc5b69f4497172b4239b1c86573f3b25430496c21de213aa" Jul 24 15:57:19.180571 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.180549 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:57:19.281694 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281653 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-certs\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281709 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-podinfo\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281737 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-credential-socket\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281769 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f672a9-5a20-4d10-b5ea-95b676127c59-istiod-ca-cert\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281788 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-envoy\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281816 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-socket\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281858 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-token\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.281899 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281890 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-data\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.282301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.281972 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pht2w\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-kube-api-access-pht2w\") pod \"68f672a9-5a20-4d10-b5ea-95b676127c59\" (UID: \"68f672a9-5a20-4d10-b5ea-95b676127c59\") " Jul 24 15:57:19.282301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282013 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:57:19.282301 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282089 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:57:19.282483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282345 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-certs\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.282483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282368 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-credential-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.282483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282384 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f672a9-5a20-4d10-b5ea-95b676127c59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:57:19.282483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282455 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:57:19.282969 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.282948 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-data" (OuterVolumeSpecName: "istio-data") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:57:19.284267 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.284225 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-token" (OuterVolumeSpecName: "istio-token") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:57:19.284378 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.284264 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-kube-api-access-pht2w" (OuterVolumeSpecName: "kube-api-access-pht2w") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "kube-api-access-pht2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:57:19.284536 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.284519 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:57:19.284601 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.284533 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68f672a9-5a20-4d10-b5ea-95b676127c59" (UID: "68f672a9-5a20-4d10-b5ea-95b676127c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:57:19.383426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383368 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68f672a9-5a20-4d10-b5ea-95b676127c59-istiod-ca-cert\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.383426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383419 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-envoy\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.383426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383430 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-workload-socket\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.383426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383438 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-token\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.383426 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383447 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-data\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.383754 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383455 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pht2w\" (UniqueName: \"kubernetes.io/projected/68f672a9-5a20-4d10-b5ea-95b676127c59-kube-api-access-pht2w\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:19.383754 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:19.383465 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68f672a9-5a20-4d10-b5ea-95b676127c59-istio-podinfo\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:20.163013 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:20.162973 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" event={"ID":"68f672a9-5a20-4d10-b5ea-95b676127c59","Type":"ContainerDied","Data":"b7681d920e98c5ab7a0d84c2673e1ee878bfda0bf1b41e3fd787d5a76b3ea441"} Jul 24 15:57:20.163013 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:20.163013 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66" Jul 24 15:57:20.163564 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:20.163048 2563 scope.go:117] "RemoveContainer" containerID="ba6c3bd09ab1a854909e93134ca7ed035cb26e4ef5b0208288c5f18634f91ac4" Jul 24 15:57:20.196132 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:20.196093 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66"] Jul 24 15:57:20.199763 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:20.199731 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-6b1d8d-openshift-default-5b9988dd78-xmw66"] Jul 24 15:57:20.437559 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:20.437477 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" path="/var/lib/kubelet/pods/68f672a9-5a20-4d10-b5ea-95b676127c59/volumes" Jul 24 15:57:33.069626 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.069581 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts"] Jul 24 15:57:33.070078 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.069788 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" podUID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" containerName="maas-api" containerID="cri-o://0d5bca2692389392e78fa5b4a875ca03813663f325a1b5b1b13fb6aebb98296b" gracePeriod=30 Jul 24 15:57:33.092454 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.092415 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:33.120926 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.120449 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx"] Jul 24 15:57:33.120926 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.120713 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" podUID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" containerName="payload-pre-processing" containerID="cri-o://2b6ce461ab4536d3274933f592c1fa2808de190b4aada9e672749cd72ca67655" gracePeriod=30 Jul 24 15:57:33.193649 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.193619 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:33.211651 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.211603 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" podUID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.70:9004: connect: connection refused" Jul 24 15:57:33.211824 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.211649 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" podUID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.69:8443/health\": dial tcp 10.132.0.69:8443: connect: connection refused" Jul 24 15:57:33.245077 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.245043 2563 generic.go:356] "Generic (PLEG): container finished" podID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" containerID="0d5bca2692389392e78fa5b4a875ca03813663f325a1b5b1b13fb6aebb98296b" exitCode=0 Jul 24 15:57:33.245309 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.245119 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" event={"ID":"f2cb84eb-4da3-42b4-9982-93e9fe116dba","Type":"ContainerDied","Data":"0d5bca2692389392e78fa5b4a875ca03813663f325a1b5b1b13fb6aebb98296b"} Jul 24 15:57:33.247387 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.247356 2563 generic.go:356] "Generic (PLEG): container finished" podID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" containerID="2b6ce461ab4536d3274933f592c1fa2808de190b4aada9e672749cd72ca67655" exitCode=0 Jul 24 15:57:33.247517 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.247435 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" event={"ID":"ade04c84-a912-46a5-a7c7-87c9f3b2838d","Type":"ContainerDied","Data":"2b6ce461ab4536d3274933f592c1fa2808de190b4aada9e672749cd72ca67655"} Jul 24 15:57:33.293597 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.292788 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:33.386365 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.386339 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:57:33.421687 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.421663 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:57:33.421996 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.421976 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb84eb-4da3-42b4-9982-93e9fe116dba-openshift-service-ca\") pod \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " Jul 24 15:57:33.422054 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.422025 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55k7c\" (UniqueName: \"kubernetes.io/projected/f2cb84eb-4da3-42b4-9982-93e9fe116dba-kube-api-access-55k7c\") pod \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " Jul 24 15:57:33.422122 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.422108 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls\") pod \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\" (UID: \"f2cb84eb-4da3-42b4-9982-93e9fe116dba\") " Jul 24 15:57:33.422416 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.422387 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cb84eb-4da3-42b4-9982-93e9fe116dba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2cb84eb-4da3-42b4-9982-93e9fe116dba" (UID: "f2cb84eb-4da3-42b4-9982-93e9fe116dba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:57:33.424372 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.424341 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2cb84eb-4da3-42b4-9982-93e9fe116dba" (UID: "f2cb84eb-4da3-42b4-9982-93e9fe116dba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:57:33.424565 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.424548 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cb84eb-4da3-42b4-9982-93e9fe116dba-kube-api-access-55k7c" (OuterVolumeSpecName: "kube-api-access-55k7c") pod "f2cb84eb-4da3-42b4-9982-93e9fe116dba" (UID: "f2cb84eb-4da3-42b4-9982-93e9fe116dba"). InnerVolumeSpecName "kube-api-access-55k7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:57:33.523347 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.523304 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dzln\" (UniqueName: \"kubernetes.io/projected/ade04c84-a912-46a5-a7c7-87c9f3b2838d-kube-api-access-2dzln\") pod \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " Jul 24 15:57:33.523559 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.523386 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade04c84-a912-46a5-a7c7-87c9f3b2838d-plugins-config-volume\") pod \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\" (UID: \"ade04c84-a912-46a5-a7c7-87c9f3b2838d\") " Jul 24 15:57:33.523685 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.523667 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2cb84eb-4da3-42b4-9982-93e9fe116dba-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:33.523750 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.523689 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55k7c\" (UniqueName: \"kubernetes.io/projected/f2cb84eb-4da3-42b4-9982-93e9fe116dba-kube-api-access-55k7c\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:33.523750 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.523704 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2cb84eb-4da3-42b4-9982-93e9fe116dba-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:33.523870 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.523784 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade04c84-a912-46a5-a7c7-87c9f3b2838d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ade04c84-a912-46a5-a7c7-87c9f3b2838d" (UID: "ade04c84-a912-46a5-a7c7-87c9f3b2838d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:57:33.525631 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.525602 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade04c84-a912-46a5-a7c7-87c9f3b2838d-kube-api-access-2dzln" (OuterVolumeSpecName: "kube-api-access-2dzln") pod "ade04c84-a912-46a5-a7c7-87c9f3b2838d" (UID: "ade04c84-a912-46a5-a7c7-87c9f3b2838d"). InnerVolumeSpecName "kube-api-access-2dzln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:57:33.624704 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.624610 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade04c84-a912-46a5-a7c7-87c9f3b2838d-plugins-config-volume\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:33.624704 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:33.624645 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzln\" (UniqueName: \"kubernetes.io/projected/ade04c84-a912-46a5-a7c7-87c9f3b2838d-kube-api-access-2dzln\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:57:34.253943 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.253914 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" Jul 24 15:57:34.254465 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.253957 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts" event={"ID":"f2cb84eb-4da3-42b4-9982-93e9fe116dba","Type":"ContainerDied","Data":"02864834423e0376ccb029c373db421c1b2b6ea308f2a9d7607d9303f6588e41"} Jul 24 15:57:34.254465 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.254029 2563 scope.go:117] "RemoveContainer" containerID="0d5bca2692389392e78fa5b4a875ca03813663f325a1b5b1b13fb6aebb98296b" Jul 24 15:57:34.255508 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.255494 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" Jul 24 15:57:34.255605 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.255486 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx" event={"ID":"ade04c84-a912-46a5-a7c7-87c9f3b2838d","Type":"ContainerDied","Data":"ce50677b146f351b50166d03ba8d298ebec0da7ab5d095a6222ce819f54ab1ba"} Jul 24 15:57:34.265346 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.265319 2563 scope.go:117] "RemoveContainer" containerID="2b6ce461ab4536d3274933f592c1fa2808de190b4aada9e672749cd72ca67655" Jul 24 15:57:34.280520 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.280490 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts"] Jul 24 15:57:34.284483 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.284456 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a64ba4-54f988789b-5fxts"] Jul 24 15:57:34.295633 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.295602 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx"] Jul 24 15:57:34.299072 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.299041 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a64ba4-85f59dc896-jwsxx"] Jul 24 15:57:34.438966 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.438862 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" path="/var/lib/kubelet/pods/ade04c84-a912-46a5-a7c7-87c9f3b2838d/volumes" Jul 24 15:57:34.439552 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:34.439526 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" path="/var/lib/kubelet/pods/f2cb84eb-4da3-42b4-9982-93e9fe116dba/volumes" Jul 24 15:57:37.613409 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:37.613363 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:37.997304 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:37.997204 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:57:43.054635 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:57:43.054593 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:06.673335 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:06.673303 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:58:06.686800 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:06.686773 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xg7m_b543d8cd-803e-4799-9c92-46c820f7dae2/ovn-acl-logging/0.log" Jul 24 15:58:19.638679 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:19.638645 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:20.235205 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:20.235167 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:21.734030 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:21.733996 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:25.805386 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:25.805349 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:39.958161 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:39.958118 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:40.058144 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.058109 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:40.239961 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.239877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:40.458034 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.457997 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:40.585134 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585089 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44"] Jul 24 15:58:40.585732 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585712 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" containerName="payload-pre-processing" Jul 24 15:58:40.585788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585741 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" containerName="payload-pre-processing" Jul 24 15:58:40.585788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585756 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" containerName="maas-api" Jul 24 15:58:40.585788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585765 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" containerName="maas-api" Jul 24 15:58:40.585788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585777 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79dbca9-6723-489d-bfb4-ded15fdcbf6c" containerName="maas-api" Jul 24 15:58:40.585788 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585785 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79dbca9-6723-489d-bfb4-ded15fdcbf6c" containerName="maas-api" Jul 24 15:58:40.585948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585808 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" Jul 24 15:58:40.585948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585816 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" Jul 24 15:58:40.585948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585828 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" Jul 24 15:58:40.585948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585837 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" Jul 24 15:58:40.585948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585858 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c791db-783c-46c5-93a3-2e1c6d76879a" containerName="payload-pre-processing" Jul 24 15:58:40.585948 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585867 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c791db-783c-46c5-93a3-2e1c6d76879a" containerName="payload-pre-processing" Jul 24 15:58:40.586146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585970 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79dbca9-6723-489d-bfb4-ded15fdcbf6c" containerName="maas-api" Jul 24 15:58:40.586146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585980 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c791db-783c-46c5-93a3-2e1c6d76879a" containerName="payload-pre-processing" Jul 24 15:58:40.586146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.585991 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cb84eb-4da3-42b4-9982-93e9fe116dba" containerName="maas-api" Jul 24 15:58:40.586146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.586002 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" Jul 24 15:58:40.586146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.586013 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f672a9-5a20-4d10-b5ea-95b676127c59" containerName="istio-proxy" Jul 24 15:58:40.586146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.586023 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ade04c84-a912-46a5-a7c7-87c9f3b2838d" containerName="payload-pre-processing" Jul 24 15:58:40.589572 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.589554 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:40.592147 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.592118 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:58:40.595358 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.595329 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44"] Jul 24 15:58:40.759869 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.759833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf090412-013b-4493-b312-619804359338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44\" (UID: \"bf090412-013b-4493-b312-619804359338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:40.861377 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.861224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf090412-013b-4493-b312-619804359338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44\" (UID: \"bf090412-013b-4493-b312-619804359338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:40.861869 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.861847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf090412-013b-4493-b312-619804359338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44\" (UID: \"bf090412-013b-4493-b312-619804359338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:40.902358 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:40.902313 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:41.045494 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:41.045469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44"] Jul 24 15:58:41.047691 ip-10-0-132-115 kubenswrapper[2563]: W0724 15:58:41.047658 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf090412_013b_4493_b312_619804359338.slice/crio-0fc0fb9220f802d0ffcdedd0ca213a5a7c6630b5651e54fc6b2877999be29c0b WatchSource:0}: Error finding container 0fc0fb9220f802d0ffcdedd0ca213a5a7c6630b5651e54fc6b2877999be29c0b: Status 404 returned error can't find the container with id 0fc0fb9220f802d0ffcdedd0ca213a5a7c6630b5651e54fc6b2877999be29c0b Jul 24 15:58:41.664427 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:41.664323 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" event={"ID":"bf090412-013b-4493-b312-619804359338","Type":"ContainerStarted","Data":"3cfb67e9a6cb3014e2d6945d4ae2209d4ccf8b7265007885f6acd6bb3bd82254"} Jul 24 15:58:41.664427 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:41.664390 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" event={"ID":"bf090412-013b-4493-b312-619804359338","Type":"ContainerStarted","Data":"0fc0fb9220f802d0ffcdedd0ca213a5a7c6630b5651e54fc6b2877999be29c0b"} Jul 24 15:58:41.682082 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:41.682017 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" podStartSLOduration=1.682000242 podStartE2EDuration="1.682000242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:58:41.680504991 +0000 UTC m=+3035.767156914" watchObservedRunningTime="2026-07-24 15:58:41.682000242 +0000 UTC m=+3035.768652154" Jul 24 15:58:42.672503 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:42.672468 2563 generic.go:356] "Generic (PLEG): container finished" podID="bf090412-013b-4493-b312-619804359338" containerID="3cfb67e9a6cb3014e2d6945d4ae2209d4ccf8b7265007885f6acd6bb3bd82254" exitCode=0 Jul 24 15:58:42.672916 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:42.672554 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" event={"ID":"bf090412-013b-4493-b312-619804359338","Type":"ContainerDied","Data":"3cfb67e9a6cb3014e2d6945d4ae2209d4ccf8b7265007885f6acd6bb3bd82254"} Jul 24 15:58:43.819942 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:43.819914 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:43.991963 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:43.991849 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf090412-013b-4493-b312-619804359338-maas-api-service-ca\") pod \"bf090412-013b-4493-b312-619804359338\" (UID: \"bf090412-013b-4493-b312-619804359338\") " Jul 24 15:58:43.992284 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:43.992234 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf090412-013b-4493-b312-619804359338-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf090412-013b-4493-b312-619804359338" (UID: "bf090412-013b-4493-b312-619804359338"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:58:44.093332 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:44.093290 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf090412-013b-4493-b312-619804359338-maas-api-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:58:44.685488 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:44.685458 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" Jul 24 15:58:44.685488 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:44.685474 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44" event={"ID":"bf090412-013b-4493-b312-619804359338","Type":"ContainerDied","Data":"0fc0fb9220f802d0ffcdedd0ca213a5a7c6630b5651e54fc6b2877999be29c0b"} Jul 24 15:58:44.685696 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:44.685519 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc0fb9220f802d0ffcdedd0ca213a5a7c6630b5651e54fc6b2877999be29c0b" Jul 24 15:58:51.157131 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.156429 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7"] Jul 24 15:58:51.157131 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.156728 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" podUID="0b564b77-c368-4b46-a097-bda5c17496fb" containerName="maas-api" containerID="cri-o://c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21" gracePeriod=30 Jul 24 15:58:51.191927 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.191877 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:51.291509 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.291470 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:51.342965 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.342930 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:51.416686 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.416619 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:58:51.458781 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.458740 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b564b77-c368-4b46-a097-bda5c17496fb-openshift-service-ca\") pod \"0b564b77-c368-4b46-a097-bda5c17496fb\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " Jul 24 15:58:51.458987 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.458805 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rptc\" (UniqueName: \"kubernetes.io/projected/0b564b77-c368-4b46-a097-bda5c17496fb-kube-api-access-9rptc\") pod \"0b564b77-c368-4b46-a097-bda5c17496fb\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " Jul 24 15:58:51.458987 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.458831 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls\") pod \"0b564b77-c368-4b46-a097-bda5c17496fb\" (UID: \"0b564b77-c368-4b46-a097-bda5c17496fb\") " Jul 24 15:58:51.459200 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.459175 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b564b77-c368-4b46-a097-bda5c17496fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b564b77-c368-4b46-a097-bda5c17496fb" (UID: "0b564b77-c368-4b46-a097-bda5c17496fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:58:51.461470 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.461439 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b564b77-c368-4b46-a097-bda5c17496fb-kube-api-access-9rptc" (OuterVolumeSpecName: "kube-api-access-9rptc") pod "0b564b77-c368-4b46-a097-bda5c17496fb" (UID: "0b564b77-c368-4b46-a097-bda5c17496fb"). InnerVolumeSpecName "kube-api-access-9rptc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:58:51.462019 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.461998 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b564b77-c368-4b46-a097-bda5c17496fb" (UID: "0b564b77-c368-4b46-a097-bda5c17496fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:58:51.559729 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.559695 2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b564b77-c368-4b46-a097-bda5c17496fb-openshift-service-ca\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:58:51.559729 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.559727 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rptc\" (UniqueName: \"kubernetes.io/projected/0b564b77-c368-4b46-a097-bda5c17496fb-kube-api-access-9rptc\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:58:51.559992 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.559745 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b564b77-c368-4b46-a097-bda5c17496fb-maas-api-tls\") on node \"ip-10-0-132-115.ec2.internal\" DevicePath \"\"" Jul 24 15:58:51.735480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.735391 2563 generic.go:356] "Generic (PLEG): container finished" podID="0b564b77-c368-4b46-a097-bda5c17496fb" containerID="c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21" exitCode=0 Jul 24 15:58:51.735480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.735434 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" Jul 24 15:58:51.735480 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.735456 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" event={"ID":"0b564b77-c368-4b46-a097-bda5c17496fb","Type":"ContainerDied","Data":"c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21"} Jul 24 15:58:51.735734 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.735499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7" event={"ID":"0b564b77-c368-4b46-a097-bda5c17496fb","Type":"ContainerDied","Data":"7ab32548dcfb6b93cdc6163f6b5f04e14e54bfce0013f369a66fd72a330f4e5c"} Jul 24 15:58:51.735734 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.735522 2563 scope.go:117] "RemoveContainer" containerID="c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21" Jul 24 15:58:51.745983 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.745960 2563 scope.go:117] "RemoveContainer" containerID="c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21" Jul 24 15:58:51.746335 ip-10-0-132-115 kubenswrapper[2563]: E0724 15:58:51.746311 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21\": container with ID starting with c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21 not found: ID does not exist" containerID="c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21" Jul 24 15:58:51.746429 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.746348 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21"} err="failed to get container status \"c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21\": rpc error: code = NotFound desc = could not find container \"c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21\": container with ID starting with c64f5549402a0cc2bc3ff893e9594b8547176e8f1cffdbf750d91958d4498d21 not found: ID does not exist" Jul 24 15:58:51.756632 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.756599 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7"] Jul 24 15:58:51.761100 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:51.761058 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4dda13-86d66454fb-p4qr7"] Jul 24 15:58:52.441534 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:52.438881 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b564b77-c368-4b46-a097-bda5c17496fb" path="/var/lib/kubelet/pods/0b564b77-c368-4b46-a097-bda5c17496fb/volumes" Jul 24 15:58:55.650443 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:55.650408 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:55.662748 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:55.662700 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44"] Jul 24 15:58:55.666201 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:55.666170 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4dda13-d7c44"] Jul 24 15:58:56.029183 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:56.029145 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:58:56.437195 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:58:56.437110 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf090412-013b-4493-b312-619804359338" path="/var/lib/kubelet/pods/bf090412-013b-4493-b312-619804359338/volumes" Jul 24 15:59:02.140453 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:02.140360 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:02.238703 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:02.238662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:02.465562 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:02.465471 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:02.707441 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:02.707407 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:13.341067 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:13.341028 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:13.436210 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:13.436179 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:13.513941 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:13.513907 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:17.920063 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:17.920022 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:18.308324 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:18.308232 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:24.827451 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:24.827408 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:24.921640 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:24.921605 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:25.103829 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:25.103735 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:25.429767 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:25.429675 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:35.972893 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:35.972854 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:36.075553 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:36.075506 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:36.172025 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:36.171988 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:40.552253 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:40.552216 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:40.945800 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:40.945716 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:47.839561 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:47.839523 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:47.954970 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:47.954932 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:48.124447 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:48.124354 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:48.509470 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:48.509441 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 15:59:54.436659 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:54.436626 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9pcbj_4eea4eb5-4f44-4e8d-b36b-6967e1597d8b/manager/0.log" Jul 24 15:59:54.563430 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:54.563383 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hztzp_779fddca-4ab7-4e51-b30d-244827b696fd/manager/0.log" Jul 24 15:59:54.681504 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:54.681473 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-549b6877d-f6dhb_be2e7525-961f-4090-9321-d583b655b8a0/manager/0.log" Jul 24 15:59:54.799734 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:54.799704 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vrdjb_faa859a6-6bbb-4318-a8fb-21b81df0c6e7/server/0.log" Jul 24 15:59:54.911181 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:54.911150 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-56ggm_96e8ac06-690b-4c08-8c7b-b9047c4f8b59/manager/0.log" Jul 24 15:59:55.026020 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:55.025994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-l8bjx_3ac15baf-c265-4ea2-9277-3fa87ba8a209/copy-manifests/0.log" Jul 24 15:59:55.032583 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:55.032561 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-l8bjx_3ac15baf-c265-4ea2-9277-3fa87ba8a209/manager/0.log" Jul 24 15:59:55.620618 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:55.620582 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59df598d7d-82bt4_022cc6e5-502e-4932-8b4f-b19de9df2127/maas-api/0.log" Jul 24 15:59:55.948863 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:55.948790 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-pn29f_09cd3231-8871-42cc-a1b2-f0826c378703/postgres/0.log" Jul 24 15:59:56.691886 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.691854 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs_2e2500b0-53b4-40d0-ba33-1b4315246a10/util/0.log" Jul 24 15:59:56.697932 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.697906 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs_2e2500b0-53b4-40d0-ba33-1b4315246a10/pull/0.log" Jul 24 15:59:56.703661 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.703640 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs_2e2500b0-53b4-40d0-ba33-1b4315246a10/extract/0.log" Jul 24 15:59:56.811801 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.811758 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh_ba270f6a-06cb-48f9-830d-96595a1e515a/util/0.log" Jul 24 15:59:56.817847 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.817819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh_ba270f6a-06cb-48f9-830d-96595a1e515a/pull/0.log" Jul 24 15:59:56.824231 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.824205 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh_ba270f6a-06cb-48f9-830d-96595a1e515a/extract/0.log" Jul 24 15:59:56.935357 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.935330 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9_6acd7206-7243-4478-a9f6-b403b597a44a/util/0.log" Jul 24 15:59:56.941216 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.941196 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9_6acd7206-7243-4478-a9f6-b403b597a44a/pull/0.log" Jul 24 15:59:56.947146 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:56.947085 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9_6acd7206-7243-4478-a9f6-b403b597a44a/extract/0.log" Jul 24 15:59:57.053604 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.053577 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv_474a2157-dd26-47d3-b13c-8f44f8cbbd81/pull/0.log" Jul 24 15:59:57.060145 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.060119 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv_474a2157-dd26-47d3-b13c-8f44f8cbbd81/extract/0.log" Jul 24 15:59:57.066585 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.066558 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv_474a2157-dd26-47d3-b13c-8f44f8cbbd81/util/0.log" Jul 24 15:59:57.291528 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.291496 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fr4h8_e368c3f7-4432-421c-af39-0f02cf74a0c7/manager/0.log" Jul 24 15:59:57.403099 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.403064 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nsd65_394da736-b485-443b-9d74-e72df7a5f17e/manager/0.log" Jul 24 15:59:57.508074 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.508045 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-f6jsv_b49eb213-b09d-40fd-871a-03f5796356b7/kuadrant-console-plugin/0.log" Jul 24 15:59:57.620557 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.620467 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jrbn9_65b1c792-36c1-4105-9b52-3645e540167f/registry-server/0.log" Jul 24 15:59:57.955501 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:57.955418 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sklnf_533263ef-432c-41af-a089-7db19363fc2b/manager/0.log" Jul 24 15:59:58.509900 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:58.509864 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-kwb8m_81080541-f392-46d6-a35d-0861c5f5aae2/kube-auth-proxy/0.log" Jul 24 15:59:58.736762 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:58.736730 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/1.log" Jul 24 15:59:58.853962 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:58.853876 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-nx4jd_60911cd6-421b-4d37-8074-59fb448465ff/payload-pre-processing/0.log" Jul 24 15:59:58.967299 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:58.967235 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hl4mb_22baeaaa-964a-44be-bc00-89064290f48c/payload-processing/0.log" Jul 24 15:59:59.412479 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:59.412444 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7spl8_d992ce95-bdfa-4a82-8dbc-4709ecc2e500/main/0.log" Jul 24 15:59:59.750785 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:59.750675 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjsw6l_a419ae67-0b6d-4616-a8d5-95dfc74b1b4b/main/0.log" Jul 24 15:59:59.856471 ip-10-0-132-115 kubenswrapper[2563]: I0724 15:59:59.856444 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fmh97_32ac9d62-1754-4271-965f-0b5480cd98ad/main/0.log" Jul 24 16:00:04.927755 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:04.927718 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 16:00:05.022183 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:05.022150 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 16:00:07.976929 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:07.976896 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wb6w7_8c1b4722-2747-4d4a-8cb6-7deca0ce9676/global-pull-secret-syncer/0.log" Jul 24 16:00:08.032738 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:08.032707 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" Jul 24 16:00:08.132010 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:08.131978 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-115.ec2.internal_a4ace6b3f39731b3e1514dbf278e40d7/haproxy/0.log" Jul 24 16:00:12.372166 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.372136 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs_2e2500b0-53b4-40d0-ba33-1b4315246a10/extract/0.log" Jul 24 16:00:12.429283 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.429229 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs_2e2500b0-53b4-40d0-ba33-1b4315246a10/util/0.log" Jul 24 16:00:12.451440 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.451398 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lm5zs_2e2500b0-53b4-40d0-ba33-1b4315246a10/pull/0.log" Jul 24 16:00:12.498024 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.497994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh_ba270f6a-06cb-48f9-830d-96595a1e515a/extract/0.log" Jul 24 16:00:12.521043 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.521017 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh_ba270f6a-06cb-48f9-830d-96595a1e515a/util/0.log" Jul 24 16:00:12.546789 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.546763 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0jxrwh_ba270f6a-06cb-48f9-830d-96595a1e515a/pull/0.log" Jul 24 16:00:12.581638 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.581605 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9_6acd7206-7243-4478-a9f6-b403b597a44a/extract/0.log" Jul 24 16:00:12.604539 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.604509 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9_6acd7206-7243-4478-a9f6-b403b597a44a/util/0.log" Jul 24 16:00:12.627938 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.627856 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73qx9x9_6acd7206-7243-4478-a9f6-b403b597a44a/pull/0.log" Jul 24 16:00:12.661915 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.661885 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv_474a2157-dd26-47d3-b13c-8f44f8cbbd81/extract/0.log" Jul 24 16:00:12.688824 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.688792 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv_474a2157-dd26-47d3-b13c-8f44f8cbbd81/util/0.log" Jul 24 16:00:12.716676 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:12.716648 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17htqv_474a2157-dd26-47d3-b13c-8f44f8cbbd81/pull/0.log" Jul 24 16:00:13.292274 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:13.292230 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-fr4h8_e368c3f7-4432-421c-af39-0f02cf74a0c7/manager/0.log" Jul 24 16:00:13.325086 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:13.325042 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nsd65_394da736-b485-443b-9d74-e72df7a5f17e/manager/0.log" Jul 24 16:00:13.360628 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:13.360598 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-f6jsv_b49eb213-b09d-40fd-871a-03f5796356b7/kuadrant-console-plugin/0.log" Jul 24 16:00:13.402983 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:13.402945 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jrbn9_65b1c792-36c1-4105-9b52-3645e540167f/registry-server/0.log" Jul 24 16:00:13.632161 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:13.632076 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sklnf_533263ef-432c-41af-a089-7db19363fc2b/manager/0.log" Jul 24 16:00:15.814348 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:15.814314 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-cx28x_94c14b7a-93f1-45ab-a38b-dc241af1d481/monitoring-plugin/0.log" Jul 24 16:00:16.032581 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:16.032548 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsvt7_12fa0c31-de73-4d43-8635-82e60062a477/node-exporter/0.log" Jul 24 16:00:16.056342 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:16.056314 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsvt7_12fa0c31-de73-4d43-8635-82e60062a477/kube-rbac-proxy/0.log" Jul 24 16:00:16.080393 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:16.080317 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsvt7_12fa0c31-de73-4d43-8635-82e60062a477/init-textfile/0.log" Jul 24 16:00:16.479072 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:16.478982 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677cf8999f-g5rgl_32912013-4bb2-466e-ba52-9e704f63b6f1/telemeter-client/0.log" Jul 24 16:00:16.501738 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:16.501707 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677cf8999f-g5rgl_32912013-4bb2-466e-ba52-9e704f63b6f1/reload/0.log" Jul 24 16:00:16.523905 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:16.523878 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-677cf8999f-g5rgl_32912013-4bb2-466e-ba52-9e704f63b6f1/kube-rbac-proxy/0.log" Jul 24 16:00:17.721234 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721196 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64"] Jul 24 16:00:17.721741 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721720 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf090412-013b-4493-b312-619804359338" containerName="revoke-keys" Jul 24 16:00:17.721741 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721742 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf090412-013b-4493-b312-619804359338" containerName="revoke-keys" Jul 24 16:00:17.721868 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721760 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b564b77-c368-4b46-a097-bda5c17496fb" containerName="maas-api" Jul 24 16:00:17.721868 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721767 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b564b77-c368-4b46-a097-bda5c17496fb" containerName="maas-api" Jul 24 16:00:17.721868 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721848 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf090412-013b-4493-b312-619804359338" containerName="revoke-keys" Jul 24 16:00:17.721868 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.721858 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b564b77-c368-4b46-a097-bda5c17496fb" containerName="maas-api" Jul 24 16:00:17.725217 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.725198 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.727644 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.727616 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5b48\"/\"openshift-service-ca.crt\"" Jul 24 16:00:17.727794 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.727645 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w5b48\"/\"default-dockercfg-qcxq2\"" Jul 24 16:00:17.728303 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.728284 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5b48\"/\"kube-root-ca.crt\"" Jul 24 16:00:17.734470 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.734445 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64"] Jul 24 16:00:17.877041 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.876998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll8b\" (UniqueName: \"kubernetes.io/projected/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-kube-api-access-mll8b\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.877041 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.877047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-sys\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.877341 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.877075 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-podres\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.877341 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.877111 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-lib-modules\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.877341 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.877169 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-proc\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978177 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978073 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll8b\" (UniqueName: \"kubernetes.io/projected/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-kube-api-access-mll8b\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978177 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978129 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-sys\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978177 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-podres\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-lib-modules\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-proc\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978309 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-sys\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978342 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-podres\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978379 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-lib-modules\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.978507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.978378 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-proc\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:17.986771 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:17.986725 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll8b\" (UniqueName: \"kubernetes.io/projected/04d8c8a4-9ce5-4925-b5e0-e575a667d4a5-kube-api-access-mll8b\") pod \"perf-node-gather-daemonset-8ks64\" (UID: \"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:18.035954 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:18.035910 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:18.187084 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:18.187053 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64"] Jul 24 16:00:18.188573 ip-10-0-132-115 kubenswrapper[2563]: W0724 16:00:18.188548 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04d8c8a4_9ce5_4925_b5e0_e575a667d4a5.slice/crio-79e38de68a618a2ea74627add6bf8e260ff3efd67ea16d5a4b48fde60c9b3a16 WatchSource:0}: Error finding container 79e38de68a618a2ea74627add6bf8e260ff3efd67ea16d5a4b48fde60c9b3a16: Status 404 returned error can't find the container with id 79e38de68a618a2ea74627add6bf8e260ff3efd67ea16d5a4b48fde60c9b3a16 Jul 24 16:00:18.253137 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:18.253104 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" event={"ID":"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5","Type":"ContainerStarted","Data":"79e38de68a618a2ea74627add6bf8e260ff3efd67ea16d5a4b48fde60c9b3a16"} Jul 24 16:00:19.206994 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:19.206960 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bdctf_393ec1dc-8487-43e5-953d-969a2b7f10aa/konnectivity-agent/0.log" limit=1024 Jul 24 16:00:19.260903 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:19.260864 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" event={"ID":"04d8c8a4-9ce5-4925-b5e0-e575a667d4a5","Type":"ContainerStarted","Data":"8f50bae687d3b18f12b134cfcffbb80df7e2e5af58e987adbd85f48f2ee806b2"} Jul 24 16:00:19.261105 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:19.261035 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:19.278576 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:19.278523 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" podStartSLOduration=2.278506851 podStartE2EDuration="2.278506851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:00:19.278294182 +0000 UTC m=+3133.364946096" watchObservedRunningTime="2026-07-24 16:00:19.278506851 +0000 UTC m=+3133.365158766" Jul 24 16:00:20.807076 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:20.807043 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9rg4_f157fb48-58c1-4f43-bc15-b8dd74dfcc02/dns/0.log" Jul 24 16:00:20.829278 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:20.829227 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9rg4_f157fb48-58c1-4f43-bc15-b8dd74dfcc02/kube-rbac-proxy/0.log" Jul 24 16:00:20.896784 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:20.896729 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-656vw_40253b47-eaff-437d-a540-2526e5e4f40d/dns-node-resolver/0.log" Jul 24 16:00:21.541843 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:21.541811 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9bzd_d5dfc302-e6b9-43f0-b89f-b264a4d5a2a0/node-ca/0.log" Jul 24 16:00:22.713087 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:22.713052 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-kwb8m_81080541-f392-46d6-a35d-0861c5f5aae2/kube-auth-proxy/0.log" Jul 24 16:00:22.770977 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:22.770935 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/0.log" Jul 24 16:00:22.796797 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:22.796761 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/1.log" Jul 24 16:00:22.830536 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:22.830502 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-nx4jd_60911cd6-421b-4d37-8074-59fb448465ff/payload-pre-processing/0.log" Jul 24 16:00:22.859918 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:22.859868 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hl4mb_22baeaaa-964a-44be-bc00-89064290f48c/payload-processing/0.log" Jul 24 16:00:23.751790 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:23.751755 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zw8rr_c3db899e-09ba-4b52-9b93-81c946f895e1/serve-healthcheck-canary/0.log" Jul 24 16:00:24.236671 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:24.236644 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jcm7r_0b5ca429-11b2-4d36-9222-8f3f71530f7d/kube-rbac-proxy/0.log" Jul 24 16:00:24.258856 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:24.258826 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jcm7r_0b5ca429-11b2-4d36-9222-8f3f71530f7d/exporter/0.log" Jul 24 16:00:24.284908 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:24.284880 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jcm7r_0b5ca429-11b2-4d36-9222-8f3f71530f7d/extractor/0.log" Jul 24 16:00:25.277052 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:25.277025 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8ks64" Jul 24 16:00:26.664831 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:26.664749 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-9pcbj_4eea4eb5-4f44-4e8d-b36b-6967e1597d8b/manager/0.log" Jul 24 16:00:26.864464 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:26.864440 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hztzp_779fddca-4ab7-4e51-b30d-244827b696fd/manager/0.log" Jul 24 16:00:26.995210 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:26.995120 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-549b6877d-f6dhb_be2e7525-961f-4090-9321-d583b655b8a0/manager/0.log" Jul 24 16:00:27.019488 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:27.019465 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vrdjb_faa859a6-6bbb-4318-a8fb-21b81df0c6e7/server/0.log" Jul 24 16:00:27.097507 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:27.097475 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-56ggm_96e8ac06-690b-4c08-8c7b-b9047c4f8b59/manager/0.log" Jul 24 16:00:27.125924 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:27.125895 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-l8bjx_3ac15baf-c265-4ea2-9277-3fa87ba8a209/manager/0.log" Jul 24 16:00:27.152158 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:27.152125 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-l8bjx_3ac15baf-c265-4ea2-9277-3fa87ba8a209/copy-manifests/0.log" Jul 24 16:00:28.704277 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:28.704172 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-kwb8m_81080541-f392-46d6-a35d-0861c5f5aae2/kube-auth-proxy/0.log" Jul 24 16:00:28.718160 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:28.718133 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-5mxs4_327c309a-2737-44ea-8002-06e8a409114a/openshift-lws-operator/0.log" Jul 24 16:00:28.735586 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:28.735545 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/0.log" Jul 24 16:00:28.764777 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:28.764743 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/1.log" Jul 24 16:00:28.790151 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:28.790123 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-nx4jd_60911cd6-421b-4d37-8074-59fb448465ff/payload-pre-processing/0.log" Jul 24 16:00:28.812813 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:28.812782 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hl4mb_22baeaaa-964a-44be-bc00-89064290f48c/payload-processing/0.log" Jul 24 16:00:30.337303 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:30.337270 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-587457bbc9-kwb8m_81080541-f392-46d6-a35d-0861c5f5aae2/kube-auth-proxy/0.log" Jul 24 16:00:30.374259 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:30.374181 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/0.log" Jul 24 16:00:30.402288 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:30.402231 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-v8z9f_bf5060b3-a279-4958-af38-7bcff776112b/istio-proxy/1.log" Jul 24 16:00:30.423304 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:30.423272 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-nx4jd_60911cd6-421b-4d37-8074-59fb448465ff/payload-pre-processing/0.log" Jul 24 16:00:30.445295 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:30.445264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-hl4mb_22baeaaa-964a-44be-bc00-89064290f48c/payload-processing/0.log" Jul 24 16:00:34.386235 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:34.386142 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f_8530a5e5-08c1-4d38-9d9c-429f9bf677c5/kube-storage-version-migrator-operator/1.log" Jul 24 16:00:34.387344 ip-10-0-132-115 kubenswrapper[2563]: I0724 16:00:34.387287 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nvw7f_8530a5e5-08c1-4d38-9d9c-429f9bf677c5/kube-storage-version-migrator-operator/0.log"