Jul 21 14:17:16.508483 ip-10-0-134-49 systemd[1]: Starting Kubernetes Kubelet... Jul 21 14:17:16.936463 ip-10-0-134-49 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:17:16.936463 ip-10-0-134-49 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 14:17:16.936463 ip-10-0-134-49 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:17:16.936463 ip-10-0-134-49 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 14:17:16.936463 ip-10-0-134-49 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 14:17:16.936463 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.933843 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938792 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938809 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938813 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938817 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938820 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938822 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938825 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938828 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938830 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938833 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938837 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938841 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938844 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938846 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938849 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938852 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938854 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938857 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938859 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:17:16.950607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938861 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938864 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938873 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938876 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938880 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938882 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938885 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938888 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938890 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938893 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938897 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938899 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938904 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938907 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938910 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938914 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938917 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938920 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938922 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938925 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:17:16.952516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938927 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938930 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938932 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938935 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938937 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938939 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938942 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938945 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938947 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938949 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938952 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938954 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938957 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938959 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938962 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938971 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938975 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938977 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938980 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938982 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:17:16.954896 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938985 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938987 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938990 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938992 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938995 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938997 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.938999 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939002 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939004 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939007 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939010 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939013 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939016 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939019 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939022 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939024 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939027 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939030 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939032 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939035 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:17:16.956986 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939037 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939042 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939046 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939049 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939052 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939054 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939057 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939060 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939064 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939066 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939069 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939071 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939074 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939076 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939079 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939081 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939084 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939086 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939089 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:17:16.959163 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939091 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939795 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939802 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939805 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939808 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939811 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939814 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939816 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939819 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939822 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939825 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939827 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939830 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939832 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939836 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939839 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939841 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939844 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939847 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939849 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:17:16.960881 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939852 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939855 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939858 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939861 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939863 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939866 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939869 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939871 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939874 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939876 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939879 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939881 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939883 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939886 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939889 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939891 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939893 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939896 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939899 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939902 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:17:16.963543 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939905 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939907 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939910 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939912 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939915 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939917 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939919 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939922 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939925 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939929 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939932 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939935 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939938 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939941 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939943 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939947 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939950 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939953 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939956 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939958 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:17:16.974774 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939961 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939964 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939966 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939969 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939972 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939974 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939977 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939979 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939982 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939985 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939989 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939993 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939996 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.939999 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940002 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940005 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940008 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940011 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940014 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:17:16.984277 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940016 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940019 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940021 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940024 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940028 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940031 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940034 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940036 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940039 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940042 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940045 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940047 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940050 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940052 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940055 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940059 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940062 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940064 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940067 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:17:17.002660 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940070 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940072 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940488 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940495 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940499 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940503 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940506 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940509 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940512 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940515 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940517 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940520 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940523 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940526 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940528 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940531 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940533 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940536 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940539 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:17:17.016574 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940542 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:17:17.012354 ip-10-0-134-49 systemd[1]: Started Kubernetes Kubelet. Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940544 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940547 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940550 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940555 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940558 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940560 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940563 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940565 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940568 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940570 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940573 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940575 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940578 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940580 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940583 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940585 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940589 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940593 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940596 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:17:17.023527 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940599 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940602 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940605 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940607 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940610 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940613 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940616 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940619 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940621 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940625 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940627 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940630 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940633 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940635 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940637 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940640 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940645 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940648 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940650 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940653 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:17:17.037589 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940656 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940658 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940661 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940663 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940666 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940668 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940673 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940676 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940678 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940681 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940683 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940687 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940690 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940692 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940695 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940698 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940700 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940703 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940705 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940708 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:17:17.233990 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940710 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940713 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940716 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940718 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940721 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940723 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940727 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940729 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940733 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940736 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940738 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940740 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940743 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940745 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940748 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940750 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940753 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940755 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940759 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940761 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:17:17.248006 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940764 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.940766 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940831 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940837 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940843 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940847 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940851 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940855 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940859 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940863 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940866 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940869 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940872 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940876 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940879 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940882 2570 flags.go:64] FLAG: --cgroup-root="" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940884 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940887 2570 flags.go:64] FLAG: --client-ca-file="" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940890 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940893 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940902 2570 flags.go:64] FLAG: --cluster-domain="" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940905 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940908 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940912 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 14:17:17.250607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940915 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940919 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940922 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940925 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940928 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940931 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940935 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940938 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940941 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940944 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940948 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940952 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940954 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940957 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940960 2570 flags.go:64] FLAG: --enable-server="true" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940963 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940967 2570 flags.go:64] FLAG: --event-burst="100" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940970 2570 flags.go:64] FLAG: --event-qps="50" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940973 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940976 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940979 2570 flags.go:64] FLAG: --eviction-hard="" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940983 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940986 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940988 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940991 2570 flags.go:64] FLAG: --eviction-soft="" Jul 21 14:17:17.253139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940994 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.940997 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941000 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941004 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941007 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941010 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941013 2570 flags.go:64] FLAG: --feature-gates="" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941017 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941020 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941023 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941026 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941029 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941032 2570 flags.go:64] FLAG: --help="false" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941036 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941039 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941042 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941045 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941048 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941051 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941054 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941056 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941059 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941063 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 14:17:17.254718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941065 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941068 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941071 2570 flags.go:64] FLAG: --kube-reserved="" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941074 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941077 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941079 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941082 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941085 2570 flags.go:64] FLAG: --lock-file="" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941088 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941090 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941093 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941098 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941102 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941105 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941108 2570 flags.go:64] FLAG: --logging-format="text" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941110 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941114 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941117 2570 flags.go:64] FLAG: --manifest-url="" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941120 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941124 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941127 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941130 2570 flags.go:64] FLAG: --max-pods="110" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941134 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941137 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941140 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 14:17:17.259436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941143 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941146 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941149 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941152 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941159 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941161 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941164 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941167 2570 flags.go:64] FLAG: --pod-cidr="" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941170 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941176 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941178 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941182 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941185 2570 flags.go:64] FLAG: --port="10250" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941188 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941190 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f830d3db02ddb02a" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941193 2570 flags.go:64] FLAG: --qos-reserved="" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941196 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941199 2570 flags.go:64] FLAG: --register-node="true" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941203 2570 flags.go:64] FLAG: --register-with-taints="" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941207 2570 flags.go:64] FLAG: --registry-burst="10" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941210 2570 flags.go:64] FLAG: --registry-qps="5" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941226 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941229 2570 flags.go:64] FLAG: --reserved-memory="" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941233 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941237 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 14:17:17.260301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941240 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941242 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941245 2570 flags.go:64] FLAG: --runonce="false" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941248 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941251 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941255 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941258 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941260 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941263 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941267 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941270 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941273 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941275 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941278 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941281 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941284 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941286 2570 flags.go:64] FLAG: --system-cgroups="" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941289 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941294 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941297 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941300 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941304 2570 flags.go:64] FLAG: --tls-min-version="" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941307 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941310 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941312 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 14:17:17.261038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941315 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941319 2570 flags.go:64] FLAG: --v="2" Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941323 2570 flags.go:64] FLAG: --version="false" Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941327 2570 flags.go:64] FLAG: --vmodule="" Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941330 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941334 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941440 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941444 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941447 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941449 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941452 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941455 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941458 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941461 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941464 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941466 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941469 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941471 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941474 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941476 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941479 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:17:17.261868 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941481 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941484 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941486 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941491 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941494 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941497 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941500 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941503 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941505 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941508 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941511 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941513 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941517 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941520 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941522 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941525 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941527 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941530 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941533 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:17:17.262517 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941536 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941538 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941541 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941543 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941547 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941550 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941552 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941555 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941557 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941560 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941562 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941564 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941567 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941570 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941572 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941574 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941577 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941580 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941582 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941584 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:17:17.263101 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941587 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941590 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941592 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941596 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941599 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941603 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941606 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941609 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941611 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941614 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941616 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941619 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941621 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941624 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941626 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941629 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941634 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941637 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941639 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:17:17.263916 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941642 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941644 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941647 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941649 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941652 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941654 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941657 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941659 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941662 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941665 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941669 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941671 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941674 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941676 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941679 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941682 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941684 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941687 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941690 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:17:17.264516 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941693 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941695 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941698 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941700 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941703 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941705 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.941708 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.941713 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.950406 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.950424 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.951521 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.951538 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951615 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951622 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951626 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:17:17.265106 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951630 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951633 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951637 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951641 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951645 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951648 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951651 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951655 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951658 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951662 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951665 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951676 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951680 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951683 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951686 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951689 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951693 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951696 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951701 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951707 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:17:17.265601 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951711 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951715 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951718 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951726 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951729 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951733 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951736 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951738 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951741 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951744 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951748 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951751 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951755 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951757 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951760 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951763 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951765 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951768 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951771 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:17:17.266209 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951774 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951777 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951780 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951782 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951786 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951788 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951791 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951794 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951796 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951799 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951801 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951804 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951806 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951809 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951811 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951813 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951819 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951822 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951824 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951827 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:17:17.266750 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951830 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951833 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951835 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951837 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951840 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951843 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951845 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951848 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951850 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951853 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951855 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951859 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951861 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951864 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951866 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951869 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951871 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951874 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951877 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:17:17.267296 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951879 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951881 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951884 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951886 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951889 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951891 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951894 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951896 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951899 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951904 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951906 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951909 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951912 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951916 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951920 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951923 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951926 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:17:17.267824 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.951928 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.951934 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952050 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952056 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952058 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952062 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952065 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952067 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952070 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952073 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952075 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952078 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952080 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952083 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952086 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 14:17:17.268289 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952088 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952091 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952093 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952096 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952098 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952101 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952103 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952106 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952108 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952112 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952116 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952119 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952122 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952124 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952127 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952131 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952135 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952137 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952140 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952143 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 14:17:17.425144 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952145 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952147 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952150 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952152 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952155 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952157 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952159 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952162 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952164 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952167 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952169 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952171 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952176 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952179 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952181 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952184 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952186 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952189 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952191 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952194 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 14:17:17.425749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952196 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952201 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952206 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952209 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952212 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952235 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952250 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952253 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952256 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952258 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952262 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952265 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952268 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952272 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952275 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952278 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952281 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952284 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952287 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 14:17:17.426375 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952290 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952292 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952295 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952298 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952300 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952303 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952306 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952309 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952311 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952314 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952317 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952319 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952322 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952324 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952326 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952331 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952334 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952336 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952339 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952343 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 14:17:17.426873 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952347 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952350 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952353 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952355 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952357 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952360 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:16.952362 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.952367 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.952519 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.956522 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.958282 2570 server.go:1019] "Starting client certificate rotation" Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.958386 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.958426 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 14:17:17.427441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.961639 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 14:17:17.427817 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.963707 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 14:17:17.427817 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.975880 2570 log.go:25] "Validated CRI v1 runtime API" Jul 21 14:17:17.427817 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.978635 2570 log.go:25] "Validated CRI v1 image API" Jul 21 14:17:17.427817 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.980346 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 14:17:17.427817 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.986290 2570 fs.go:135] Filesystem UUIDs: map[21aea43d-8dc8-46ed-baed-0f069e3c25ce:/dev/nvme0n1p3 6c7afa65-a1bf-4327-98ea-0880f6f840f3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 21 14:17:17.427817 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.986306 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.991540 2570 manager.go:216] Machine: {Timestamp:2026-07-21 14:17:16.990366905 +0000 UTC m=+0.365666506 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199974 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21fbc0ebd89ff0a54f9f9d65305868 SystemUUID:ec21fbc0-ebd8-9ff0-a54f-9f9d65305868 BootID:67156372-7481-494b-b046-6818809576e5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b7:c2:59:cd:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b7:c2:59:cd:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:31:94:9c:32:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.991685 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.991779 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.993332 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.993357 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-49.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.994649 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.994661 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.994669 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.995714 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.997072 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:16.999294 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000682 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000794 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000822 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000832 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000847 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 14:17:17.428174 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000856 2570 kubelet.go:398] "Adding apiserver pod source" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.000865 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.001213 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.001242 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.003781 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.006035 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.006062 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007395 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007411 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007416 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007421 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007430 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007435 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007443 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007448 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007455 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007460 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007468 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.007473 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.008332 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.008340 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.011456 2570 server.go:1268] "Started kubelet" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.011565 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.011549 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.011623 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.012057 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-49.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.012668 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.012670 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-49.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.013472 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.013628 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.016975 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-49.ec2.internal.18c453c021cc7510 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-49.ec2.internal,UID:ip-10-0-134-49.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-49.ec2.internal,},FirstTimestamp:2026-07-21 14:17:17.01143272 +0000 UTC m=+0.386732321,LastTimestamp:2026-07-21 14:17:17.01143272 +0000 UTC m=+0.386732321,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-49.ec2.internal,}" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.019775 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.020870 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.020888 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.020974 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.021538 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.021552 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.021658 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.021692 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.023036 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.025785 2570 factory.go:55] Registering systemd factory Jul 21 14:17:17.428698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.025801 2570 factory.go:223] Registration of the systemd container factory successfully Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.026007 2570 factory.go:153] Registering CRI-O factory Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.026019 2570 factory.go:223] Registration of the crio container factory successfully Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.026069 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.026092 2570 factory.go:103] Registering Raw factory Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.026101 2570 manager.go:1196] Started watching for new ooms in manager Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.026488 2570 manager.go:349] Starting recovery of all containers Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.030561 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-49.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.030677 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.036301 2570 manager.go:354] Recovery completed Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.041673 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045152 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045180 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045190 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045931 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045953 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045973 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.045950 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.046792 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.046816 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.046825 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.047497 2570 policy_none.go:49] "None policy: Start" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.047515 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.047528 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.047786 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-49.ec2.internal.18c453c023cf2e6e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-49.ec2.internal,UID:ip-10-0-134-49.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-49.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-49.ec2.internal,},FirstTimestamp:2026-07-21 14:17:17.045165678 +0000 UTC m=+0.420465279,LastTimestamp:2026-07-21 14:17:17.045165678 +0000 UTC m=+0.420465279,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-49.ec2.internal,}" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.048689 2570 policy_none.go:47] "Start" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.052576 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8lpct" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.057762 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-49.ec2.internal.18c453c023cf77d7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-49.ec2.internal,UID:ip-10-0-134-49.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-49.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-49.ec2.internal,},FirstTimestamp:2026-07-21 14:17:17.045184471 +0000 UTC m=+0.420484072,LastTimestamp:2026-07-21 14:17:17.045184471 +0000 UTC m=+0.420484072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-49.ec2.internal,}" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.060892 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8lpct" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102323 2570 manager.go:337] "Starting Device Plugin manager" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.102368 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102381 2570 server.go:87] "Starting device plugin registration server" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102612 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 14:17:17.429884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102621 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102745 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102813 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.102820 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.103286 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.103347 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.125472 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.126613 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.126644 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.126661 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.126694 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.128899 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.203624 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.204816 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.204845 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.204859 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.204888 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.213469 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.213492 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-49.ec2.internal\": node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.227562 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal"] Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.227631 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.228364 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.228381 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.228390 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.229621 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.229762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.229783 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.230375 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.230394 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.230402 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.230468 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.230490 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.230503 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.231420 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.231446 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.232021 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.232036 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.232045 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.249007 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-49.ec2.internal\" not found" node="ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.254359 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-49.ec2.internal\" not found" node="ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.430761 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.275318 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.324596 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2e3692874c3c91f11b8448272247d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal\" (UID: \"b2e3692874c3c91f11b8448272247d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.324618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2e3692874c3c91f11b8448272247d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal\" (UID: \"b2e3692874c3c91f11b8448272247d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.324635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62ca7a90d9db4064d93d60a128f27362-config\") pod \"kube-apiserver-proxy-ip-10-0-134-49.ec2.internal\" (UID: \"62ca7a90d9db4064d93d60a128f27362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.376306 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.425742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2e3692874c3c91f11b8448272247d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal\" (UID: \"b2e3692874c3c91f11b8448272247d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.425767 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2e3692874c3c91f11b8448272247d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal\" (UID: \"b2e3692874c3c91f11b8448272247d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.425784 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62ca7a90d9db4064d93d60a128f27362-config\") pod \"kube-apiserver-proxy-ip-10-0-134-49.ec2.internal\" (UID: \"62ca7a90d9db4064d93d60a128f27362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.425835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62ca7a90d9db4064d93d60a128f27362-config\") pod \"kube-apiserver-proxy-ip-10-0-134-49.ec2.internal\" (UID: \"62ca7a90d9db4064d93d60a128f27362\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.425856 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2e3692874c3c91f11b8448272247d50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal\" (UID: \"b2e3692874c3c91f11b8448272247d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.431873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.425835 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2e3692874c3c91f11b8448272247d50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal\" (UID: \"b2e3692874c3c91f11b8448272247d50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.486428 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.476984 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.552257 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.551654 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.663889 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:17.557510 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" Jul 21 14:17:17.663889 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.577678 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.779814 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.678099 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:17.779814 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.778640 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:18.959686 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:17.879056 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:19.087561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:18.965385 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 14:17:19.087561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:18.965517 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 14:17:19.087561 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:18.987935 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:19.087561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.072710 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 14:12:18 +0000 UTC" deadline="2028-01-15 21:16:05.393900725 +0000 UTC" Jul 21 14:17:19.087561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.072740 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13038h58m46.321164454s" Jul 21 14:17:19.089977 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:19.089853 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.125881 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.142404 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.143379 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientMemory" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.143410 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.143421 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeHasSufficientPID" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:19.145799 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ca7a90d9db4064d93d60a128f27362.slice/crio-92a30708580b5972eb6b3f9ca16df0e1174d78d30fcc306294d1124c351c274c WatchSource:0}: Error finding container 92a30708580b5972eb6b3f9ca16df0e1174d78d30fcc306294d1124c351c274c: Status 404 returned error can't find the container with id 92a30708580b5972eb6b3f9ca16df0e1174d78d30fcc306294d1124c351c274c Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:19.146110 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e3692874c3c91f11b8448272247d50.slice/crio-c67ecee389dbe9ceac49a585f8d0a24a880dda4451e628f923f43f90c821067b WatchSource:0}: Error finding container c67ecee389dbe9ceac49a585f8d0a24a880dda4451e628f923f43f90c821067b: Status 404 returned error can't find the container with id c67ecee389dbe9ceac49a585f8d0a24a880dda4451e628f923f43f90c821067b Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.150319 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:19.190509 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-49.ec2.internal\" not found" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.192350 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.229497 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.231984 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.244785 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.245752 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" Jul 21 14:17:19.277854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:19.252592 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 14:17:20.008958 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.008930 2570 apiserver.go:52] "Watching apiserver" Jul 21 14:17:20.019217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.019194 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 14:17:20.020747 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.020727 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-7t9g8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal","openshift-multus/network-metrics-daemon-rdkx6","kube-system/global-pull-secret-syncer-p447f","kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal","openshift-multus/multus-additional-cni-plugins-xmsqf","openshift-multus/multus-g9dbn","openshift-network-diagnostics/network-check-target-mr7zj","openshift-network-operator/iptables-alerter-ph2tg","openshift-ovn-kubernetes/ovnkube-node-wbx42","kube-system/konnectivity-agent-9725k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm","openshift-cluster-node-tuning-operator/tuned-zsg8t","openshift-dns/node-resolver-g442s"] Jul 21 14:17:20.022249 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.022233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.023783 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.023762 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.024864 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.024844 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 14:17:20.024967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.024848 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.024967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.024897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.024967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.024947 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.025119 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.024958 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:20.025119 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.025098 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tkhdk\"" Jul 21 14:17:20.025184 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.025140 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 14:17:20.026020 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.025989 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.026102 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.026067 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:20.026214 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.026200 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 14:17:20.026352 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.026338 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w9dtm\"" Jul 21 14:17:20.026425 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.026374 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 14:17:20.027166 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.027146 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.029633 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.028506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.031467 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.031150 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.031578 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.031504 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sk9kn\"" Jul 21 14:17:20.031578 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.031569 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 14:17:20.031959 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.031942 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 14:17:20.032121 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.031969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.032760 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.032742 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 14:17:20.032922 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.032906 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.033166 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.033147 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.033794 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.033757 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-59mf9\"" Jul 21 14:17:20.034001 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.033981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:20.034106 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.034060 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:20.034787 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.034768 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.034991 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.034977 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.035382 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.035360 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-889tm\"" Jul 21 14:17:20.035532 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.035365 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.035795 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.035780 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.036134 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.036104 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.036591 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.036573 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t6swq\"" Jul 21 14:17:20.036674 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.036644 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 14:17:20.037255 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.037225 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.038225 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.038175 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 14:17:20.038304 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.038293 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 14:17:20.038521 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.038396 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 14:17:20.038617 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.038540 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.038617 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.038546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.038754 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.038742 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.039601 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.039582 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mv2w8\"" Jul 21 14:17:20.040177 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.040157 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 14:17:20.041298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041179 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.041298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041232 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zsh5h\"" Jul 21 14:17:20.041456 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041304 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.041524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041503 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 14:17:20.041910 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041845 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n6k44\"" Jul 21 14:17:20.041984 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041935 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 14:17:20.041984 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041957 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-socket-dir-parent\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042088 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041998 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 14:17:20.042088 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.041845 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 14:17:20.042088 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042040 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjp2\" (UniqueName: \"kubernetes.io/projected/b7c4a248-b6b8-4e1a-a350-217535cbcc17-kube-api-access-dbjp2\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042088 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-system-cni-dir\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.042244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042098 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.042244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6779bd01-50d9-4eaa-8770-e6ad4699153e-iptables-alerter-script\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.042244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042153 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtk5b\" (UniqueName: \"kubernetes.io/projected/6779bd01-50d9-4eaa-8770-e6ad4699153e-kube-api-access-xtk5b\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.042244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-cni-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042221 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7c4a248-b6b8-4e1a-a350-217535cbcc17-cni-binary-copy\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-hostroot\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f9a0b12-10a7-4628-92ed-9d9651940451-kubelet-config\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/292c96a4-7f66-4fca-adfc-8a86176f342a-hosts-file\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042346 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-os-release\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-netns\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-daemon-config\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmq9\" (UniqueName: \"kubernetes.io/projected/25040ae5-5c9b-42fd-9271-d2b3e472162d-kube-api-access-6vmq9\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.042466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042453 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042482 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7mc\" (UniqueName: \"kubernetes.io/projected/0730acd0-6273-431f-a01f-b5eefe741362-kube-api-access-4g7mc\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6779bd01-50d9-4eaa-8770-e6ad4699153e-host-slash\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-multus-certs\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-etc-kubernetes\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042608 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90b15efc-d945-4740-9546-757b99ec15d2-agent-certs\") pod \"konnectivity-agent-9725k\" (UID: \"90b15efc-d945-4740-9546-757b99ec15d2\") " pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-k8s-cni-cncf-io\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042653 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042689 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f9a0b12-10a7-4628-92ed-9d9651940451-dbus\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042721 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.042773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjph7\" (UniqueName: \"kubernetes.io/projected/292c96a4-7f66-4fca-adfc-8a86176f342a-kube-api-access-sjph7\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-system-cni-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-cni-multus\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-conf-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042857 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-cnibin\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-os-release\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042923 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90b15efc-d945-4740-9546-757b99ec15d2-konnectivity-ca\") pod \"konnectivity-agent-9725k\" (UID: \"90b15efc-d945-4740-9546-757b99ec15d2\") " pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.042994 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-kubelet\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.043045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292c96a4-7f66-4fca-adfc-8a86176f342a-tmp-dir\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.043076 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-cnibin\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.043201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.043097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-cni-bin\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.047447 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.047425 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 14:17:20.073033 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.072991 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxkcg" Jul 21 14:17:20.081232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.081214 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kxkcg" Jul 21 14:17:20.130945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.130918 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 14:17:20.138622 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.138578 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" event={"ID":"b2e3692874c3c91f11b8448272247d50","Type":"ContainerStarted","Data":"c67ecee389dbe9ceac49a585f8d0a24a880dda4451e628f923f43f90c821067b"} Jul 21 14:17:20.139682 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.139658 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" event={"ID":"62ca7a90d9db4064d93d60a128f27362","Type":"ContainerStarted","Data":"92a30708580b5972eb6b3f9ca16df0e1174d78d30fcc306294d1124c351c274c"} Jul 21 14:17:20.143888 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-system-cni-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.143998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-cni-multus\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.143998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143913 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-conf-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.143998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90b15efc-d945-4740-9546-757b99ec15d2-konnectivity-ca\") pod \"konnectivity-agent-9725k\" (UID: \"90b15efc-d945-4740-9546-757b99ec15d2\") " pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.143998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-registration-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.143998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-host\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.143995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-cni-multus\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-kubelet\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144031 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-conf-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144049 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-system-cni-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144073 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-kubelet\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-systemd\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144167 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-cnibin\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144188 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-lib-modules\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjp2\" (UniqueName: \"kubernetes.io/projected/b7c4a248-b6b8-4e1a-a350-217535cbcc17-kube-api-access-dbjp2\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144224 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-cnibin\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-system-cni-dir\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.144261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtk5b\" (UniqueName: \"kubernetes.io/projected/6779bd01-50d9-4eaa-8770-e6ad4699153e-kube-api-access-xtk5b\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-node-log\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144323 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysconfig\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144277 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-system-cni-dir\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144343 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-sys\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144378 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx49b\" (UniqueName: \"kubernetes.io/projected/8e764b6e-4080-4374-a52e-de405b446221-kube-api-access-nx49b\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-cni-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7c4a248-b6b8-4e1a-a350-217535cbcc17-cni-binary-copy\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-hostroot\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144476 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/292c96a4-7f66-4fca-adfc-8a86176f342a-hosts-file\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144502 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovnkube-config\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144528 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltn8w\" (UniqueName: \"kubernetes.io/projected/f91b710d-ac22-48bd-b037-439391d77258-kube-api-access-ltn8w\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144533 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-hostroot\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-netns\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144570 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/292c96a4-7f66-4fca-adfc-8a86176f342a-hosts-file\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmq9\" (UniqueName: \"kubernetes.io/projected/25040ae5-5c9b-42fd-9271-d2b3e472162d-kube-api-access-6vmq9\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-netns\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.144822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144628 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/90b15efc-d945-4740-9546-757b99ec15d2-konnectivity-ca\") pod \"konnectivity-agent-9725k\" (UID: \"90b15efc-d945-4740-9546-757b99ec15d2\") " pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7mc\" (UniqueName: \"kubernetes.io/projected/0730acd0-6273-431f-a01f-b5eefe741362-kube-api-access-4g7mc\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.144844 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-socket-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.144941 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:20.644917812 +0000 UTC m=+3.012427837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-log-socket\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.144962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7c4a248-b6b8-4e1a-a350-217535cbcc17-cni-binary-copy\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-kubernetes\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145064 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysctl-d\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-etc-kubernetes\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145122 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-etc-kubernetes\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-var-lib-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145186 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-var-lib-kubelet\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-cnibin\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.145634 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.145231 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-os-release\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-cni-dir\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145270 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-cnibin\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.145291 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:20.645277092 +0000 UTC m=+3.012787100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145293 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-os-release\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145313 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145347 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-sys-fs\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-cni-netd\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145439 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f91b710d-ac22-48bd-b037-439391d77258-serviceca\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145472 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292c96a4-7f66-4fca-adfc-8a86176f342a-tmp-dir\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145501 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-kubelet\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145523 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-run-netns\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-cni-bin\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-modprobe-d\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145672 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysctl-conf\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145694 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-run\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145720 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-cni-bin\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145750 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145779 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-var-lib-cni-bin\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292c96a4-7f66-4fca-adfc-8a86176f342a-tmp-dir\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145829 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e764b6e-4080-4374-a52e-de405b446221-etc-tuned\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-socket-dir-parent\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6779bd01-50d9-4eaa-8770-e6ad4699153e-iptables-alerter-script\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-socket-dir-parent\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145967 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.145993 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-slash\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146032 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-systemd\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.146930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-env-overrides\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f9a0b12-10a7-4628-92ed-9d9651940451-kubelet-config\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146106 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25040ae5-5c9b-42fd-9271-d2b3e472162d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-device-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-ovn\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1f9a0b12-10a7-4628-92ed-9d9651940451-kubelet-config\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146190 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovn-node-metrics-cert\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146214 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovnkube-script-lib\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9l8\" (UniqueName: \"kubernetes.io/projected/87804a09-c932-4e91-a618-cf5a5a1d9a86-kube-api-access-js9l8\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146262 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b710d-ac22-48bd-b037-439391d77258-host\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-os-release\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146352 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-daemon-config\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146366 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-os-release\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146383 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6779bd01-50d9-4eaa-8770-e6ad4699153e-iptables-alerter-script\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6779bd01-50d9-4eaa-8770-e6ad4699153e-host-slash\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146418 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6779bd01-50d9-4eaa-8770-e6ad4699153e-host-slash\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.147552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-systemd-units\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-etc-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146568 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e764b6e-4080-4374-a52e-de405b446221-tmp\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-multus-certs\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90b15efc-d945-4740-9546-757b99ec15d2-agent-certs\") pod \"konnectivity-agent-9725k\" (UID: \"90b15efc-d945-4740-9546-757b99ec15d2\") " pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-multus-certs\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146762 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnp9m\" (UniqueName: \"kubernetes.io/projected/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-kube-api-access-nnp9m\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-k8s-cni-cncf-io\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146807 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7c4a248-b6b8-4e1a-a350-217535cbcc17-multus-daemon-config\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146860 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f9a0b12-10a7-4628-92ed-9d9651940451-dbus\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjph7\" (UniqueName: \"kubernetes.io/projected/292c96a4-7f66-4fca-adfc-8a86176f342a-kube-api-access-sjph7\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.146943 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.147093 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7c4a248-b6b8-4e1a-a350-217535cbcc17-host-run-k8s-cni-cncf-io\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.147168 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1f9a0b12-10a7-4628-92ed-9d9651940451-dbus\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.148111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.147223 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25040ae5-5c9b-42fd-9271-d2b3e472162d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.150484 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.150463 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/90b15efc-d945-4740-9546-757b99ec15d2-agent-certs\") pod \"konnectivity-agent-9725k\" (UID: \"90b15efc-d945-4740-9546-757b99ec15d2\") " pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.156333 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.156279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjp2\" (UniqueName: \"kubernetes.io/projected/b7c4a248-b6b8-4e1a-a350-217535cbcc17-kube-api-access-dbjp2\") pod \"multus-g9dbn\" (UID: \"b7c4a248-b6b8-4e1a-a350-217535cbcc17\") " pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.156517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.156495 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtk5b\" (UniqueName: \"kubernetes.io/projected/6779bd01-50d9-4eaa-8770-e6ad4699153e-kube-api-access-xtk5b\") pod \"iptables-alerter-ph2tg\" (UID: \"6779bd01-50d9-4eaa-8770-e6ad4699153e\") " pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.156930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.156885 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7mc\" (UniqueName: \"kubernetes.io/projected/0730acd0-6273-431f-a01f-b5eefe741362-kube-api-access-4g7mc\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.157092 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.157070 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmq9\" (UniqueName: \"kubernetes.io/projected/25040ae5-5c9b-42fd-9271-d2b3e472162d-kube-api-access-6vmq9\") pod \"multus-additional-cni-plugins-xmsqf\" (UID: \"25040ae5-5c9b-42fd-9271-d2b3e472162d\") " pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.157702 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.157685 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjph7\" (UniqueName: \"kubernetes.io/projected/292c96a4-7f66-4fca-adfc-8a86176f342a-kube-api-access-sjph7\") pod \"node-resolver-g442s\" (UID: \"292c96a4-7f66-4fca-adfc-8a86176f342a\") " pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.247982 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.247944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:20.247982 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.247987 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-sys-fs\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-cni-netd\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f91b710d-ac22-48bd-b037-439391d77258-serviceca\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248135 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-sys-fs\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248164 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-etc-selinux\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248142 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-cni-netd\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248141 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-kubelet\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248193 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-kubelet\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-run-netns\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-cni-bin\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-run-netns\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-modprobe-d\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysctl-conf\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-cni-bin\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248314 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-run\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e764b6e-4080-4374-a52e-de405b446221-etc-tuned\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248374 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-run\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-modprobe-d\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-slash\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-run-ovn-kubernetes\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248437 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-systemd\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248445 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysctl-conf\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248457 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-slash\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-env-overrides\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.248641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248451 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248475 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-systemd\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-device-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-ovn\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248556 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovn-node-metrics-cert\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248569 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-device-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248578 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovnkube-script-lib\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248588 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-ovn\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248595 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js9l8\" (UniqueName: \"kubernetes.io/projected/87804a09-c932-4e91-a618-cf5a5a1d9a86-kube-api-access-js9l8\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b710d-ac22-48bd-b037-439391d77258-host\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248631 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f91b710d-ac22-48bd-b037-439391d77258-serviceca\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-systemd-units\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248696 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-etc-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e764b6e-4080-4374-a52e-de405b446221-tmp\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.249481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnp9m\" (UniqueName: \"kubernetes.io/projected/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-kube-api-access-nnp9m\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248838 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-registration-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248863 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-host\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-systemd\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248914 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-run-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f91b710d-ac22-48bd-b037-439391d77258-host\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-etc-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-lib-modules\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-node-log\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249051 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-lib-modules\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249058 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysconfig\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249077 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-sys\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249104 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx49b\" (UniqueName: \"kubernetes.io/projected/8e764b6e-4080-4374-a52e-de405b446221-kube-api-access-nx49b\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249107 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-host\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.248887 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-env-overrides\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249131 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovnkube-config\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-systemd\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltn8w\" (UniqueName: \"kubernetes.io/projected/f91b710d-ac22-48bd-b037-439391d77258-kube-api-access-ltn8w\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.250232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249179 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-systemd-units\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-socket-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249213 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-registration-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249219 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-log-socket\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-log-socket\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249261 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-kubernetes\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysctl-d\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249299 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-kubernetes\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-var-lib-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249361 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-var-lib-kubelet\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249395 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysctl-d\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249105 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovnkube-script-lib\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249431 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-var-lib-kubelet\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249471 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-var-lib-openvswitch\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249509 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87804a09-c932-4e91-a618-cf5a5a1d9a86-node-log\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249512 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-etc-sysconfig\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249573 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8e764b6e-4080-4374-a52e-de405b446221-sys\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.250936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249601 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-socket-dir\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.251626 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.249845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovnkube-config\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.251626 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.251098 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87804a09-c932-4e91-a618-cf5a5a1d9a86-ovn-node-metrics-cert\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.251708 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.251620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e764b6e-4080-4374-a52e-de405b446221-tmp\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.251864 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.251844 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8e764b6e-4080-4374-a52e-de405b446221-etc-tuned\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.255640 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.255618 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:20.255640 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.255647 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:20.255810 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.255660 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:20.255810 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.255717 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:17:20.755699522 +0000 UTC m=+3.123209531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:20.258084 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.258061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx49b\" (UniqueName: \"kubernetes.io/projected/8e764b6e-4080-4374-a52e-de405b446221-kube-api-access-nx49b\") pod \"tuned-zsg8t\" (UID: \"8e764b6e-4080-4374-a52e-de405b446221\") " pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.258867 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.258845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnp9m\" (UniqueName: \"kubernetes.io/projected/3cd008ca-1c47-4a69-8e17-c5dd1581b9ad-kube-api-access-nnp9m\") pod \"aws-ebs-csi-driver-node-2gnkm\" (UID: \"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.259104 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.259062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltn8w\" (UniqueName: \"kubernetes.io/projected/f91b710d-ac22-48bd-b037-439391d77258-kube-api-access-ltn8w\") pod \"node-ca-7t9g8\" (UID: \"f91b710d-ac22-48bd-b037-439391d77258\") " pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.259104 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.259093 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9l8\" (UniqueName: \"kubernetes.io/projected/87804a09-c932-4e91-a618-cf5a5a1d9a86-kube-api-access-js9l8\") pod \"ovnkube-node-wbx42\" (UID: \"87804a09-c932-4e91-a618-cf5a5a1d9a86\") " pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.333683 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.333649 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g9dbn" Jul 21 14:17:20.341725 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.341700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:20.351502 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.351472 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g442s" Jul 21 14:17:20.356237 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.356215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" Jul 21 14:17:20.362853 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.362833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ph2tg" Jul 21 14:17:20.369506 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.369486 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" Jul 21 14:17:20.375447 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.375426 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:20.380900 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.380880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7t9g8" Jul 21 14:17:20.385353 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.385336 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" Jul 21 14:17:20.404247 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.404225 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91b710d_ac22_48bd_b037_439391d77258.slice/crio-09eea41ae744fe7d8d0d062ed632ddb02c497fa0fed0acac19325f0401918793 WatchSource:0}: Error finding container 09eea41ae744fe7d8d0d062ed632ddb02c497fa0fed0acac19325f0401918793: Status 404 returned error can't find the container with id 09eea41ae744fe7d8d0d062ed632ddb02c497fa0fed0acac19325f0401918793 Jul 21 14:17:20.406274 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.406206 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25040ae5_5c9b_42fd_9271_d2b3e472162d.slice/crio-6c397ea7ba4145e68f35a419b3dac3d83038969df0c1e57a8a3c0f810cb861b9 WatchSource:0}: Error finding container 6c397ea7ba4145e68f35a419b3dac3d83038969df0c1e57a8a3c0f810cb861b9: Status 404 returned error can't find the container with id 6c397ea7ba4145e68f35a419b3dac3d83038969df0c1e57a8a3c0f810cb861b9 Jul 21 14:17:20.407506 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.407439 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6779bd01_50d9_4eaa_8770_e6ad4699153e.slice/crio-d521303a6fadcc1b5259ed37e29b537d17e02ef6121169712d495133869e6418 WatchSource:0}: Error finding container d521303a6fadcc1b5259ed37e29b537d17e02ef6121169712d495133869e6418: Status 404 returned error can't find the container with id d521303a6fadcc1b5259ed37e29b537d17e02ef6121169712d495133869e6418 Jul 21 14:17:20.411470 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.411327 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd008ca_1c47_4a69_8e17_c5dd1581b9ad.slice/crio-754da9c787e0aa5b7b0e17870e571084f29863b9699a02f5f1c68a1620a0f59c WatchSource:0}: Error finding container 754da9c787e0aa5b7b0e17870e571084f29863b9699a02f5f1c68a1620a0f59c: Status 404 returned error can't find the container with id 754da9c787e0aa5b7b0e17870e571084f29863b9699a02f5f1c68a1620a0f59c Jul 21 14:17:20.413691 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.413650 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292c96a4_7f66_4fca_adfc_8a86176f342a.slice/crio-c0d640cb8074247570ea3651ba932f4deb6e042480721d50b58625a7d3bb166e WatchSource:0}: Error finding container c0d640cb8074247570ea3651ba932f4deb6e042480721d50b58625a7d3bb166e: Status 404 returned error can't find the container with id c0d640cb8074247570ea3651ba932f4deb6e042480721d50b58625a7d3bb166e Jul 21 14:17:20.414867 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.414841 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e764b6e_4080_4374_a52e_de405b446221.slice/crio-73ed8e0d7d69fa2a959c195ea2729adb23c1140c500d6384db0ae6686d9d2997 WatchSource:0}: Error finding container 73ed8e0d7d69fa2a959c195ea2729adb23c1140c500d6384db0ae6686d9d2997: Status 404 returned error can't find the container with id 73ed8e0d7d69fa2a959c195ea2729adb23c1140c500d6384db0ae6686d9d2997 Jul 21 14:17:20.415712 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.415590 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87804a09_c932_4e91_a618_cf5a5a1d9a86.slice/crio-386aac3017e17d5a771145536ed00e88b4d0b4e7c8f8bd7dd42116815436ef2d WatchSource:0}: Error finding container 386aac3017e17d5a771145536ed00e88b4d0b4e7c8f8bd7dd42116815436ef2d: Status 404 returned error can't find the container with id 386aac3017e17d5a771145536ed00e88b4d0b4e7c8f8bd7dd42116815436ef2d Jul 21 14:17:20.416684 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.416665 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b15efc_d945_4740_9546_757b99ec15d2.slice/crio-f43053c493bc4a3a364e23b8e7269ac5549e87b12c8875e24025bf5de3b21f83 WatchSource:0}: Error finding container f43053c493bc4a3a364e23b8e7269ac5549e87b12c8875e24025bf5de3b21f83: Status 404 returned error can't find the container with id f43053c493bc4a3a364e23b8e7269ac5549e87b12c8875e24025bf5de3b21f83 Jul 21 14:17:20.417501 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:20.417394 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c4a248_b6b8_4e1a_a350_217535cbcc17.slice/crio-9017603a0f75144aae54663d3e86f1986b1716468590fe27ba56ea4204c61fd6 WatchSource:0}: Error finding container 9017603a0f75144aae54663d3e86f1986b1716468590fe27ba56ea4204c61fd6: Status 404 returned error can't find the container with id 9017603a0f75144aae54663d3e86f1986b1716468590fe27ba56ea4204c61fd6 Jul 21 14:17:20.519882 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.519818 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 14:17:20.652053 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.652032 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:20.652131 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.652066 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:20.652200 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.652188 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:20.652267 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.652238 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:21.652221539 +0000 UTC m=+4.019731551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:20.652318 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.652272 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:20.652351 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.652325 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:21.652309363 +0000 UTC m=+4.019819383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:20.854476 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:20.854391 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:20.854597 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.854541 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:20.854597 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.854564 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:20.854597 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.854577 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:20.854752 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:20.854640 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:17:21.854620525 +0000 UTC m=+4.222130535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:21.083274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.083211 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 14:12:20 +0000 UTC" deadline="2028-01-15 10:28:24.0838603 +0000 UTC" Jul 21 14:17:21.083274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.083277 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13028h11m3.000587016s" Jul 21 14:17:21.147710 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.146847 2570 generic.go:356] "Generic (PLEG): container finished" podID="b2e3692874c3c91f11b8448272247d50" containerID="79f7bbb23b35767deb96af2946dd2ccbd19742533e8f1f5a2554a6f49e694f6c" exitCode=0 Jul 21 14:17:21.147710 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.146924 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" event={"ID":"b2e3692874c3c91f11b8448272247d50","Type":"ContainerDied","Data":"79f7bbb23b35767deb96af2946dd2ccbd19742533e8f1f5a2554a6f49e694f6c"} Jul 21 14:17:21.151070 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.150980 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" event={"ID":"8e764b6e-4080-4374-a52e-de405b446221","Type":"ContainerStarted","Data":"73ed8e0d7d69fa2a959c195ea2729adb23c1140c500d6384db0ae6686d9d2997"} Jul 21 14:17:21.153219 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.153115 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"386aac3017e17d5a771145536ed00e88b4d0b4e7c8f8bd7dd42116815436ef2d"} Jul 21 14:17:21.161628 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.161604 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" event={"ID":"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad","Type":"ContainerStarted","Data":"754da9c787e0aa5b7b0e17870e571084f29863b9699a02f5f1c68a1620a0f59c"} Jul 21 14:17:21.164382 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.164357 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g442s" event={"ID":"292c96a4-7f66-4fca-adfc-8a86176f342a","Type":"ContainerStarted","Data":"c0d640cb8074247570ea3651ba932f4deb6e042480721d50b58625a7d3bb166e"} Jul 21 14:17:21.169569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.169508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ph2tg" event={"ID":"6779bd01-50d9-4eaa-8770-e6ad4699153e","Type":"ContainerStarted","Data":"d521303a6fadcc1b5259ed37e29b537d17e02ef6121169712d495133869e6418"} Jul 21 14:17:21.170928 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.170879 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerStarted","Data":"6c397ea7ba4145e68f35a419b3dac3d83038969df0c1e57a8a3c0f810cb861b9"} Jul 21 14:17:21.173694 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.173635 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g9dbn" event={"ID":"b7c4a248-b6b8-4e1a-a350-217535cbcc17","Type":"ContainerStarted","Data":"9017603a0f75144aae54663d3e86f1986b1716468590fe27ba56ea4204c61fd6"} Jul 21 14:17:21.179292 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.179229 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9725k" event={"ID":"90b15efc-d945-4740-9546-757b99ec15d2","Type":"ContainerStarted","Data":"f43053c493bc4a3a364e23b8e7269ac5549e87b12c8875e24025bf5de3b21f83"} Jul 21 14:17:21.184878 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.184820 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7t9g8" event={"ID":"f91b710d-ac22-48bd-b037-439391d77258","Type":"ContainerStarted","Data":"09eea41ae744fe7d8d0d062ed632ddb02c497fa0fed0acac19325f0401918793"} Jul 21 14:17:21.661656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.660836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:21.661656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.661109 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:21.661656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.661122 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:21.661656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.661176 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:23.66115784 +0000 UTC m=+6.028667860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:21.661656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.661225 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:21.661656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.661271 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:23.661256748 +0000 UTC m=+6.028766757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:21.863135 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:21.863105 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:21.863305 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.863275 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:21.863305 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.863295 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:21.863442 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.863307 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:21.863442 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:21.863359 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:17:23.863341582 +0000 UTC m=+6.230851591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:22.083634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:22.083527 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 14:12:20 +0000 UTC" deadline="2028-04-21 01:46:11.06185427 +0000 UTC" Jul 21 14:17:22.083634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:22.083561 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15347h28m48.978297122s" Jul 21 14:17:22.137830 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:22.137187 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:22.137830 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:22.137319 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:22.137830 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:22.137684 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:22.144309 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:22.137771 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:22.144309 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:22.144128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:22.144309 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:22.144256 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:22.220034 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:22.219757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" event={"ID":"b2e3692874c3c91f11b8448272247d50","Type":"ContainerStarted","Data":"458edf8d823314bb2858166103be10183511a5af86a147cb5e4f74b106398e16"} Jul 21 14:17:23.678829 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:23.678792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:23.679286 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:23.678844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:23.679286 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.678995 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:23.679286 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.679072 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:27.679055691 +0000 UTC m=+10.046565697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:23.679286 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.678997 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:23.679286 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.679145 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:27.679127107 +0000 UTC m=+10.046637128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:23.880411 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:23.880376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:23.880592 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.880542 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:23.880592 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.880571 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:23.880592 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.880587 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:23.880740 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:23.880652 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:17:27.88063223 +0000 UTC m=+10.248142237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:24.136714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:24.135190 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:24.136714 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:24.135312 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:24.136714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:24.135202 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:24.136714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:24.135363 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:24.136714 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:24.135435 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:24.136714 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:24.135820 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:26.135486 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:26.135238 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:26.135486 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:26.135280 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:26.135486 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:26.135367 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:26.135486 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:26.135450 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:26.136100 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:26.135516 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:26.136100 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:26.135584 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:27.714283 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:27.714249 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:27.714759 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:27.714299 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:27.714759 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.714413 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:27.714759 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.714466 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:35.714450143 +0000 UTC m=+18.081960149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:27.714759 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.714411 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:27.714759 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.714553 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:35.714534155 +0000 UTC m=+18.082044165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:27.915537 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:27.915510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:27.916098 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.915670 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:27.916098 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.915694 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:27.916098 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.915708 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:27.916098 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:27.915759 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:17:35.915742871 +0000 UTC m=+18.283252881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:28.135892 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:28.135426 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:28.135892 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:28.135550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:28.135892 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:28.135559 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:28.135892 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:28.135656 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:28.135892 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:28.135709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:28.135892 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:28.135775 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:30.135595 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:30.135551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:30.136048 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:30.135563 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:30.136048 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:30.135691 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:30.136048 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:30.135723 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:30.136048 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:30.135826 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:30.136048 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:30.135899 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:32.135709 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:32.135674 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:32.136143 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:32.135795 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:32.136143 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:32.135674 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:32.136143 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:32.135674 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:32.136143 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:32.135904 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:32.136143 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:32.135980 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:34.135118 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:34.135031 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:34.135118 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:34.135055 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:34.135118 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:34.135031 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:34.135656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:34.135172 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:34.135656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:34.135324 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:34.135656 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:34.135396 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:35.774270 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:35.774230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:35.774739 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:35.774278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:35.774739 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.774405 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:35.774739 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.774410 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:35.774739 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.774466 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:51.774448537 +0000 UTC m=+34.141958555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:35.774739 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.774484 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:51.774475778 +0000 UTC m=+34.141985788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:35.976198 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:35.976157 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:35.976364 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.976344 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:35.976439 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.976376 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:35.976439 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.976392 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:35.976528 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:35.976456 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:17:51.976437614 +0000 UTC m=+34.343947636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:36.134991 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:36.134912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:36.135180 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:36.134917 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:36.135180 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:36.135052 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:36.135180 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:36.135064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:36.135180 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:36.135169 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:36.135360 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:36.135253 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:38.135530 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:38.135462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:38.136035 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:38.135583 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:38.136035 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:38.135606 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:38.136035 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:38.135581 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:38.136035 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:38.135698 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:38.136035 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:38.135789 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:40.135308 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.134936 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:40.136076 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.134969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:40.136076 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:40.135368 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:40.136076 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.134936 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:40.136076 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:40.135396 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:40.136076 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:40.135501 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:40.260311 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.260277 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7t9g8" event={"ID":"f91b710d-ac22-48bd-b037-439391d77258","Type":"ContainerStarted","Data":"c6b5fa4b4b1ee567b1ebd88e2d47f2a91256c1b4ca7595f135c7e7c5ffcda862"} Jul 21 14:17:40.261637 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.261606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" event={"ID":"62ca7a90d9db4064d93d60a128f27362","Type":"ContainerStarted","Data":"5d4ee83d1ccfb46b20f930d33611927d0fc128ad430f0b529647ee800bef2df2"} Jul 21 14:17:40.262926 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.262895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" event={"ID":"8e764b6e-4080-4374-a52e-de405b446221","Type":"ContainerStarted","Data":"90c03df8c6a53546021a6c7cb6460c711d8cd6b86ec932d324a65f25172652f7"} Jul 21 14:17:40.268025 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.267986 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"234bc7195086ba563a0135c3c226ae7a34d4d59e6a543dad88ab267963cecc0a"} Jul 21 14:17:40.268142 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.268098 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"8b0d9f5806007015886f2eb316703ff0c16980f0c9a218cbf4470a46375b1854"} Jul 21 14:17:40.268203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.268148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"569124fd629b5aa05a2614105a9c92f3902577b129237ad4b53d9df8a6d20f23"} Jul 21 14:17:40.268203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.268194 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"a1ec42b9e8b447ebc7f69a447f15f1260e582d481c9537e5eb37f8a9b1bec65c"} Jul 21 14:17:40.268303 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.268239 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"95be76e241b1a378c3de6df1c6b175fee721bd2b92f118f1764ae3a60838c70c"} Jul 21 14:17:40.268303 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.268284 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"f0b5847aace268cf183a3c55497187142b807a6e8f3d2598f8ac901d58f8d43a"} Jul 21 14:17:40.269291 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.269270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" event={"ID":"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad","Type":"ContainerStarted","Data":"fc4c8eda8b914046d73a43d420f323e92ef203ce634b2ddbb8da71d3358de78b"} Jul 21 14:17:40.270397 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.270377 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g442s" event={"ID":"292c96a4-7f66-4fca-adfc-8a86176f342a","Type":"ContainerStarted","Data":"13afd1a5fef7d11d0cc0fa7a96770537cd16cc0ad020730a6167876e2f682544"} Jul 21 14:17:40.271651 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.271630 2570 generic.go:356] "Generic (PLEG): container finished" podID="25040ae5-5c9b-42fd-9271-d2b3e472162d" containerID="2224694b2c8150b11435c06f8cec70c7851e28aa0db294c402ba7ba22d785b5c" exitCode=0 Jul 21 14:17:40.271764 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.271691 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerDied","Data":"2224694b2c8150b11435c06f8cec70c7851e28aa0db294c402ba7ba22d785b5c"} Jul 21 14:17:40.273114 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.273092 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g9dbn" event={"ID":"b7c4a248-b6b8-4e1a-a350-217535cbcc17","Type":"ContainerStarted","Data":"de77cde71d0ec0598bfd17eae3330daa10098933585d0353b81af48b464c9d2a"} Jul 21 14:17:40.274444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.274425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9725k" event={"ID":"90b15efc-d945-4740-9546-757b99ec15d2","Type":"ContainerStarted","Data":"3d74be00997b347d3341f15e4edf22e6c71ed5400fd2a526566fa5b12bc4955d"} Jul 21 14:17:40.280524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.280483 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-49.ec2.internal" podStartSLOduration=21.280473369 podStartE2EDuration="21.280473369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:22.239205313 +0000 UTC m=+4.606715342" watchObservedRunningTime="2026-07-21 14:17:40.280473369 +0000 UTC m=+22.647983397" Jul 21 14:17:40.306775 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.306738 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7t9g8" podStartSLOduration=3.543782285 podStartE2EDuration="22.306726227s" totalImagesPullingTime="18.762943942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:40.280797009 +0000 UTC m=+22.648307036" watchObservedRunningTime="2026-07-21 14:17:40.306726227 +0000 UTC m=+22.674236254" Jul 21 14:17:40.343456 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.343403 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-49.ec2.internal" podStartSLOduration=21.343383766 podStartE2EDuration="21.343383766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:40.322661446 +0000 UTC m=+22.690171474" watchObservedRunningTime="2026-07-21 14:17:40.343383766 +0000 UTC m=+22.710893795" Jul 21 14:17:40.343617 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.343591 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g9dbn" podStartSLOduration=3.437440254 podStartE2EDuration="22.343584206s" totalImagesPullingTime="18.906143952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:40.343145229 +0000 UTC m=+22.710655258" watchObservedRunningTime="2026-07-21 14:17:40.343584206 +0000 UTC m=+22.711094238" Jul 21 14:17:40.360438 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.360398 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g442s" podStartSLOduration=11.985070671999999 podStartE2EDuration="22.360382959s" totalImagesPullingTime="10.375312287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:40.35973373 +0000 UTC m=+22.727243760" watchObservedRunningTime="2026-07-21 14:17:40.360382959 +0000 UTC m=+22.727892988" Jul 21 14:17:40.381227 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:40.381176 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zsg8t" podStartSLOduration=3.510212401 podStartE2EDuration="22.381156955s" totalImagesPullingTime="18.870944554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:40.38081931 +0000 UTC m=+22.748329339" watchObservedRunningTime="2026-07-21 14:17:40.381156955 +0000 UTC m=+22.748666985" Jul 21 14:17:41.037656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.037632 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 14:17:41.120485 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.120394 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T14:17:41.03765067Z","UUID":"9cca4bda-2ba7-4419-bc8b-1d6be0fbed04","Handler":null,"Name":"","Endpoint":""} Jul 21 14:17:41.121743 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.121726 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 14:17:41.121854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.121760 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 14:17:41.278220 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.278148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" event={"ID":"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad","Type":"ContainerStarted","Data":"8cd30bd705510a594da6ff82067adc44cd26586ae30c63be390668d297ee280e"} Jul 21 14:17:41.279554 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.279506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ph2tg" event={"ID":"6779bd01-50d9-4eaa-8770-e6ad4699153e","Type":"ContainerStarted","Data":"58c759a4d2db6121318c4be48711e2549bf262dc0bc68c9e894a3d3bc0550bbc"} Jul 21 14:17:41.296762 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.296714 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9725k" podStartSLOduration=4.434043128 podStartE2EDuration="23.296701905s" totalImagesPullingTime="18.862658777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:40.400509853 +0000 UTC m=+22.768019905" watchObservedRunningTime="2026-07-21 14:17:41.296701905 +0000 UTC m=+23.664211932" Jul 21 14:17:41.297278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.297248 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ph2tg" podStartSLOduration=12.91593916 podStartE2EDuration="23.297239715s" totalImagesPullingTime="10.381300555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:41.296582498 +0000 UTC m=+23.664092525" watchObservedRunningTime="2026-07-21 14:17:41.297239715 +0000 UTC m=+23.664749742" Jul 21 14:17:41.343276 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.343243 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:41.344072 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:41.344050 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:42.135681 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:42.135636 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:42.135874 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:42.135750 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:42.135874 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:42.135645 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:42.135874 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:42.135839 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:42.135874 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:42.135642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:42.136159 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:42.135922 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:42.285944 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:42.285850 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"4a5ebd4f208f85fd631cc8d0337fe3f248038337cef8c75ff5f45571fe5949e6"} Jul 21 14:17:42.286389 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:42.286108 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:42.286685 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:42.286669 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9725k" Jul 21 14:17:44.135227 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:44.135040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:44.135657 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:44.135034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:44.135657 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:44.135325 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:44.135657 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:44.135404 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:44.135657 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:44.135043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:44.135657 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:44.135508 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:46.134780 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.134742 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:46.135531 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.134742 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:46.135531 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:46.134878 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:46.135531 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.134887 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:46.135531 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:46.134984 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:46.135531 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:46.135096 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:46.296679 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.296649 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" event={"ID":"87804a09-c932-4e91-a618-cf5a5a1d9a86","Type":"ContainerStarted","Data":"160bed68fc021d90e41fbab96c37c2a38c467a477903124a8f1320abad78c62b"} Jul 21 14:17:46.296995 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.296839 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:46.298476 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.298452 2570 generic.go:356] "Generic (PLEG): container finished" podID="25040ae5-5c9b-42fd-9271-d2b3e472162d" containerID="116929e74c726759021639d809a54af0c55e110f3217a8916a5f4bb8339e1a5e" exitCode=0 Jul 21 14:17:46.298604 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.298487 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerDied","Data":"116929e74c726759021639d809a54af0c55e110f3217a8916a5f4bb8339e1a5e"} Jul 21 14:17:46.311625 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.311605 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:46.331979 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:46.331940 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" podStartSLOduration=9.397506684 podStartE2EDuration="28.331928101s" totalImagesPullingTime="18.934421417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:46.331345657 +0000 UTC m=+28.698855697" watchObservedRunningTime="2026-07-21 14:17:46.331928101 +0000 UTC m=+28.699438128" Jul 21 14:17:47.303837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.303805 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" event={"ID":"3cd008ca-1c47-4a69-8e17-c5dd1581b9ad","Type":"ContainerStarted","Data":"0a66e2b0b38477a3c08a779af813ab32e276069c2bbbf4cd1db5a1d2cfdcbffc"} Jul 21 14:17:47.304339 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.304018 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:47.304418 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.304399 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:47.320048 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.320026 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:17:47.333846 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.333803 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2gnkm" podStartSLOduration=3.771458827 podStartE2EDuration="29.333787605s" totalImagesPullingTime="25.562328778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:17:47.333461108 +0000 UTC m=+29.700971136" watchObservedRunningTime="2026-07-21 14:17:47.333787605 +0000 UTC m=+29.701297637" Jul 21 14:17:47.471291 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.471254 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdkx6"] Jul 21 14:17:47.471457 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.471410 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:47.471553 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:47.471528 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:47.474225 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.474202 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mr7zj"] Jul 21 14:17:47.474421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.474281 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:47.474421 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:47.474343 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:47.482533 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.482496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p447f"] Jul 21 14:17:47.482649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:47.482603 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:47.482719 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:47.482700 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:48.308212 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:48.308179 2570 generic.go:356] "Generic (PLEG): container finished" podID="25040ae5-5c9b-42fd-9271-d2b3e472162d" containerID="804b9c148750935e47bb21c016854d0f3d3d30459f73e37d608a69c23bbdc831" exitCode=0 Jul 21 14:17:48.308627 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:48.308271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerDied","Data":"804b9c148750935e47bb21c016854d0f3d3d30459f73e37d608a69c23bbdc831"} Jul 21 14:17:49.135034 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:49.134987 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:49.135186 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:49.135018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:49.135186 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:49.135104 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:49.135186 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:49.135112 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:49.135186 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:49.135165 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:49.135421 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:49.135274 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:50.314952 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:50.314746 2570 generic.go:356] "Generic (PLEG): container finished" podID="25040ae5-5c9b-42fd-9271-d2b3e472162d" containerID="2d8fb128b19c5bb0d408a31e35f0677e117c4ec354cafdbb396543258fd22a0e" exitCode=0 Jul 21 14:17:50.314952 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:50.314822 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerDied","Data":"2d8fb128b19c5bb0d408a31e35f0677e117c4ec354cafdbb396543258fd22a0e"} Jul 21 14:17:51.135287 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.135258 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:51.135454 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.135345 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:51.135454 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.135351 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p447f" podUID="1f9a0b12-10a7-4628-92ed-9d9651940451" Jul 21 14:17:51.135454 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.135368 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:51.135454 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.135431 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr7zj" podUID="1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e" Jul 21 14:17:51.135613 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.135504 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rdkx6" podUID="0730acd0-6273-431f-a01f-b5eefe741362" Jul 21 14:17:51.794019 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.793959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:51.794436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.794033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:51.794436 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.794113 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:51.794436 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.794181 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:51.794436 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.794187 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs podName:0730acd0-6273-431f-a01f-b5eefe741362 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.794168385 +0000 UTC m=+66.161678411 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs") pod "network-metrics-daemon-rdkx6" (UID: "0730acd0-6273-431f-a01f-b5eefe741362") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 14:17:51.794436 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.794233 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret podName:1f9a0b12-10a7-4628-92ed-9d9651940451 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.79421724 +0000 UTC m=+66.161727250 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret") pod "global-pull-secret-syncer-p447f" (UID: "1f9a0b12-10a7-4628-92ed-9d9651940451") : object "kube-system"/"original-pull-secret" not registered Jul 21 14:17:51.995776 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.995745 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:51.995928 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.995894 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 14:17:51.995928 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.995913 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 14:17:51.995928 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.995924 2570 projected.go:196] Error preparing data for projected volume kube-api-access-2lfj9 for pod openshift-network-diagnostics/network-check-target-mr7zj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:51.996056 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:51.995971 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9 podName:1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.995959048 +0000 UTC m=+66.363469054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2lfj9" (UniqueName: "kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9") pod "network-check-target-mr7zj" (UID: "1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 14:17:51.997370 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.997353 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-49.ec2.internal" event="NodeReady" Jul 21 14:17:51.997468 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:51.997450 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 14:17:52.036236 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.036156 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-kg4q9"] Jul 21 14:17:52.038142 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.038095 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv"] Jul 21 14:17:52.038379 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.038340 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.041042 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.040963 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:17:52.042852 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.042057 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 14:17:52.042852 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.042069 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 14:17:52.042852 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.042549 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fvhtp\"" Jul 21 14:17:52.042852 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.042734 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 14:17:52.044172 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.044147 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-846d7479f5-jpdkd"] Jul 21 14:17:52.044512 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.044494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.046383 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.046364 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd"] Jul 21 14:17:52.046507 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.046486 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.047201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.046997 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 14:17:52.047296 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.047224 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 14:17:52.047296 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.047283 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 14:17:52.047524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.047416 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ct6sk\"" Jul 21 14:17:52.047802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.047783 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:17:52.048117 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.048098 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fcf8cfd48-sjz45"] Jul 21 14:17:52.048279 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.048210 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.049233 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.049211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 14:17:52.049382 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.049236 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 14:17:52.049450 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.049438 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 14:17:52.049503 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.049477 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2s4f2\"" Jul 21 14:17:52.049575 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.049511 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 14:17:52.050286 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.050265 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7"] Jul 21 14:17:52.050903 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.050447 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.050903 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.050460 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:17:52.051718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.051483 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 14:17:52.051847 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.051824 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 14:17:52.051990 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.051971 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4b5fp\"" Jul 21 14:17:52.052261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.051987 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 14:17:52.052825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.052512 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68"] Jul 21 14:17:52.052825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.052710 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" Jul 21 14:17:52.055292 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.055276 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-kg4q9"] Jul 21 14:17:52.055381 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.055333 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv"] Jul 21 14:17:52.055689 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.055386 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.056417 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.056398 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wnph8\"" Jul 21 14:17:52.056515 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.056460 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 14:17:52.056673 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.056640 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 14:17:52.057330 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.057220 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68"] Jul 21 14:17:52.057740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.057720 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 14:17:52.057940 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.057925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-87nvz\"" Jul 21 14:17:52.058075 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.058062 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-76pfl"] Jul 21 14:17:52.059991 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.059875 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 14:17:52.061019 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.060987 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 14:17:52.061106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.061076 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xp7xl"] Jul 21 14:17:52.061287 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.061233 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.063329 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.063310 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.063438 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.063419 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 14:17:52.063813 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.063796 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 14:17:52.063902 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.063848 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rkkbj\"" Jul 21 14:17:52.065648 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.065627 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s5hdl\"" Jul 21 14:17:52.065742 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.065683 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 14:17:52.065850 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.065822 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 14:17:52.065991 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.065969 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 14:17:52.068909 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.068848 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd"] Jul 21 14:17:52.070157 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.070113 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846d7479f5-jpdkd"] Jul 21 14:17:52.075905 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.075885 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fcf8cfd48-sjz45"] Jul 21 14:17:52.078542 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.078500 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-76pfl"] Jul 21 14:17:52.090287 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.090251 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7"] Jul 21 14:17:52.093120 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.093098 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xp7xl"] Jul 21 14:17:52.196706 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.196671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s4ls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-kube-api-access-2s4ls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.196871 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.196854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtfg\" (UniqueName: \"kubernetes.io/projected/2d60d6e8-4ce0-43ce-a0c2-0ced19251643-kube-api-access-cmtfg\") pod \"network-check-source-7bc5b6bf85-hqml7\" (UID: \"2d60d6e8-4ce0-43ce-a0c2-0ced19251643\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" Jul 21 14:17:52.196925 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.196885 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-ca-trust-extracted\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.196964 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.196930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-bound-sa-token\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197037 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.196960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9f27890-994f-491e-b14e-52319993fc85-cert\") pod \"ingress-canary-xp7xl\" (UID: \"d9f27890-994f-491e-b14e-52319993fc85\") " pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.197037 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197019 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-image-registry-private-configuration\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.197137 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197064 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-certificates\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.197137 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.197216 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/808270b1-884c-43ed-ab2e-4f9ab3102332-ca-trust-extracted\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197216 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.197280 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197234 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxct7\" (UniqueName: \"kubernetes.io/projected/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-kube-api-access-zxct7\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.197280 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-trusted-ca\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197342 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf728902-c923-4e7a-aeb1-94338a8b1d95-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.197342 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-bound-sa-token\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.197401 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-installation-pull-secrets\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.197401 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197389 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hkw\" (UniqueName: \"kubernetes.io/projected/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-kube-api-access-h8hkw\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.197474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-serving-cert\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.197474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197440 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-serving-cert\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.197474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-certificates\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197565 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.197565 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444c9d9-48ba-4131-9b16-2c8e26296df0-tmp-dir\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.197565 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-config\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.197690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197587 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-installation-pull-secrets\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197623 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dlvb\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-kube-api-access-9dlvb\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197674 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zx2\" (UniqueName: \"kubernetes.io/projected/d9f27890-994f-491e-b14e-52319993fc85-kube-api-access-x8zx2\") pod \"ingress-canary-xp7xl\" (UID: \"d9f27890-994f-491e-b14e-52319993fc85\") " pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.197823 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197724 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-trusted-ca\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.197823 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lvqq\" (UniqueName: \"kubernetes.io/projected/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-kube-api-access-7lvqq\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.197823 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197790 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-image-registry-private-configuration\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.197823 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.197986 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.197986 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e444c9d9-48ba-4131-9b16-2c8e26296df0-config-volume\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.197986 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.197980 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.198141 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.198021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-trusted-ca\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.198141 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.198097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxh89\" (UniqueName: \"kubernetes.io/projected/e444c9d9-48ba-4131-9b16-2c8e26296df0-kube-api-access-vxh89\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.198141 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.198133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-config\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.299458 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299368 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxh89\" (UniqueName: \"kubernetes.io/projected/e444c9d9-48ba-4131-9b16-2c8e26296df0-kube-api-access-vxh89\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.299458 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-config\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.299458 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s4ls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-kube-api-access-2s4ls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299466 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtfg\" (UniqueName: \"kubernetes.io/projected/2d60d6e8-4ce0-43ce-a0c2-0ced19251643-kube-api-access-cmtfg\") pod \"network-check-source-7bc5b6bf85-hqml7\" (UID: \"2d60d6e8-4ce0-43ce-a0c2-0ced19251643\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299508 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-ca-trust-extracted\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299552 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-bound-sa-token\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9f27890-994f-491e-b14e-52319993fc85-cert\") pod \"ingress-canary-xp7xl\" (UID: \"d9f27890-994f-491e-b14e-52319993fc85\") " pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-image-registry-private-configuration\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299612 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-certificates\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299664 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/808270b1-884c-43ed-ab2e-4f9ab3102332-ca-trust-extracted\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.299730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxct7\" (UniqueName: \"kubernetes.io/projected/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-kube-api-access-zxct7\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.299747 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.299806 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert podName:bf728902-c923-4e7a-aeb1-94338a8b1d95 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:52.799788995 +0000 UTC m=+35.167299008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jvz68" (UID: "bf728902-c923-4e7a-aeb1-94338a8b1d95") : secret "networking-console-plugin-cert" not found Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299750 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-trusted-ca\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299918 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-ca-trust-extracted\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299943 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf728902-c923-4e7a-aeb1-94338a8b1d95-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299968 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-bound-sa-token\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.299995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-installation-pull-secrets\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.300035 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300043 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hkw\" (UniqueName: \"kubernetes.io/projected/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-kube-api-access-h8hkw\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300073 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-serving-cert\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.300087 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls podName:e444c9d9-48ba-4131-9b16-2c8e26296df0 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:52.800068719 +0000 UTC m=+35.167578729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls") pod "dns-default-76pfl" (UID: "e444c9d9-48ba-4131-9b16-2c8e26296df0") : secret "dns-default-metrics-tls" not found Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-serving-cert\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.300168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300153 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-certificates\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300188 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444c9d9-48ba-4131-9b16-2c8e26296df0-tmp-dir\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-config\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300275 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-installation-pull-secrets\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dlvb\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-kube-api-access-9dlvb\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300314 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-config\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zx2\" (UniqueName: \"kubernetes.io/projected/d9f27890-994f-491e-b14e-52319993fc85-kube-api-access-x8zx2\") pod \"ingress-canary-xp7xl\" (UID: \"d9f27890-994f-491e-b14e-52319993fc85\") " pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300385 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-trusted-ca\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lvqq\" (UniqueName: \"kubernetes.io/projected/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-kube-api-access-7lvqq\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300442 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-image-registry-private-configuration\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e444c9d9-48ba-4131-9b16-2c8e26296df0-config-volume\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-trusted-ca\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.300686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-trusted-ca\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.301297 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300668 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/808270b1-884c-43ed-ab2e-4f9ab3102332-ca-trust-extracted\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.301297 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.300687 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf728902-c923-4e7a-aeb1-94338a8b1d95-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.301297 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.300745 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:52.301297 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.300760 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fcf8cfd48-sjz45: secret "image-registry-tls" not found Jul 21 14:17:52.301297 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.300800 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls podName:d7e477da-8762-4d12-b6ad-7cfceec3dd1f nodeName:}" failed. No retries permitted until 2026-07-21 14:17:52.800786253 +0000 UTC m=+35.168296281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls") pod "image-registry-fcf8cfd48-sjz45" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f") : secret "image-registry-tls" not found Jul 21 14:17:52.301499 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.301353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.301567 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-trusted-ca\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.302354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-trusted-ca\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.302844 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-config\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.302876 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e444c9d9-48ba-4131-9b16-2c8e26296df0-config-volume\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.302961 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.302975 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d7479f5-jpdkd: secret "image-registry-tls" not found Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.303034 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls podName:808270b1-884c-43ed-ab2e-4f9ab3102332 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:52.803019452 +0000 UTC m=+35.170529472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls") pod "image-registry-846d7479f5-jpdkd" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332") : secret "image-registry-tls" not found Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.303267 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e444c9d9-48ba-4131-9b16-2c8e26296df0-tmp-dir\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.303343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.303304 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-certificates\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.303836 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.303718 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-certificates\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.305074 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.304995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-installation-pull-secrets\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.305074 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.304995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-image-registry-private-configuration\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.305250 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.305095 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.305250 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.305144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9f27890-994f-491e-b14e-52319993fc85-cert\") pod \"ingress-canary-xp7xl\" (UID: \"d9f27890-994f-491e-b14e-52319993fc85\") " pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.305350 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.305274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-installation-pull-secrets\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.305494 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.305477 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-serving-cert\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.305552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.305523 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-image-registry-private-configuration\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.306478 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.306459 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-serving-cert\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.310975 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.310955 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtfg\" (UniqueName: \"kubernetes.io/projected/2d60d6e8-4ce0-43ce-a0c2-0ced19251643-kube-api-access-cmtfg\") pod \"network-check-source-7bc5b6bf85-hqml7\" (UID: \"2d60d6e8-4ce0-43ce-a0c2-0ced19251643\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" Jul 21 14:17:52.311127 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.311106 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-bound-sa-token\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.311333 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.311318 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxh89\" (UniqueName: \"kubernetes.io/projected/e444c9d9-48ba-4131-9b16-2c8e26296df0-kube-api-access-vxh89\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.312883 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.312856 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-bound-sa-token\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.313539 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.313495 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dlvb\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-kube-api-access-9dlvb\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.313637 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.313590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hkw\" (UniqueName: \"kubernetes.io/projected/5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2-kube-api-access-h8hkw\") pod \"service-ca-operator-5bcbc94c78-kdxvv\" (UID: \"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.314310 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.314288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxct7\" (UniqueName: \"kubernetes.io/projected/795f58c3-8685-44d3-a1f0-bf1a3bed6ca6-kube-api-access-zxct7\") pod \"kube-storage-version-migrator-operator-7658d9b749-5d5xd\" (UID: \"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.314837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.314700 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s4ls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-kube-api-access-2s4ls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.314974 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.314955 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zx2\" (UniqueName: \"kubernetes.io/projected/d9f27890-994f-491e-b14e-52319993fc85-kube-api-access-x8zx2\") pod \"ingress-canary-xp7xl\" (UID: \"d9f27890-994f-491e-b14e-52319993fc85\") " pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.315090 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.315073 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lvqq\" (UniqueName: \"kubernetes.io/projected/f7f3c68c-9553-4b61-a33f-c30d4e11f9c1-kube-api-access-7lvqq\") pod \"console-operator-565b7b569d-kg4q9\" (UID: \"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1\") " pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.354641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.354617 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:17:52.362639 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.362618 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" Jul 21 14:17:52.384079 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.384047 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" Jul 21 14:17:52.400100 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.399845 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" Jul 21 14:17:52.422356 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.420682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xp7xl" Jul 21 14:17:52.538997 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.538824 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-kg4q9"] Jul 21 14:17:52.544749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:52.544716 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f3c68c_9553_4b61_a33f_c30d4e11f9c1.slice/crio-5d0a4ecca3232f158076d2d0b0a425978e1359da9261283f62e0b1854c50329a WatchSource:0}: Error finding container 5d0a4ecca3232f158076d2d0b0a425978e1359da9261283f62e0b1854c50329a: Status 404 returned error can't find the container with id 5d0a4ecca3232f158076d2d0b0a425978e1359da9261283f62e0b1854c50329a Jul 21 14:17:52.556084 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.556054 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv"] Jul 21 14:17:52.560550 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:52.560520 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0aa87c_bc2b_4b5c_93e8_11d991b31fd2.slice/crio-2e34411014f7f72706bdbb40fc523468a535837b158c92417180b2f7038b6e1f WatchSource:0}: Error finding container 2e34411014f7f72706bdbb40fc523468a535837b158c92417180b2f7038b6e1f: Status 404 returned error can't find the container with id 2e34411014f7f72706bdbb40fc523468a535837b158c92417180b2f7038b6e1f Jul 21 14:17:52.575890 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.575867 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd"] Jul 21 14:17:52.578553 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:52.578525 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795f58c3_8685_44d3_a1f0_bf1a3bed6ca6.slice/crio-6f06cd34fdf52a83c2bc8c14babcefa8a116c2f877fd80016984079d96dc211c WatchSource:0}: Error finding container 6f06cd34fdf52a83c2bc8c14babcefa8a116c2f877fd80016984079d96dc211c: Status 404 returned error can't find the container with id 6f06cd34fdf52a83c2bc8c14babcefa8a116c2f877fd80016984079d96dc211c Jul 21 14:17:52.585553 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.585474 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7"] Jul 21 14:17:52.588370 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:52.588339 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d60d6e8_4ce0_43ce_a0c2_0ced19251643.slice/crio-d75f7fe529353d270c568d7515bf50daff6d5da738c72f6bcacccc7ebc4fd259 WatchSource:0}: Error finding container d75f7fe529353d270c568d7515bf50daff6d5da738c72f6bcacccc7ebc4fd259: Status 404 returned error can't find the container with id d75f7fe529353d270c568d7515bf50daff6d5da738c72f6bcacccc7ebc4fd259 Jul 21 14:17:52.604126 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.604105 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xp7xl"] Jul 21 14:17:52.606859 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:17:52.606836 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f27890_994f_491e_b14e_52319993fc85.slice/crio-d43581a3fba38c5a54743c95b46babc2ceb92fc2ef7da58e82caca4ee082d0c5 WatchSource:0}: Error finding container d43581a3fba38c5a54743c95b46babc2ceb92fc2ef7da58e82caca4ee082d0c5: Status 404 returned error can't find the container with id d43581a3fba38c5a54743c95b46babc2ceb92fc2ef7da58e82caca4ee082d0c5 Jul 21 14:17:52.804491 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.804418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:52.804491 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.804460 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804570 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804584 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d7479f5-jpdkd: secret "image-registry-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804583 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804604 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fcf8cfd48-sjz45: secret "image-registry-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.804611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804640 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls podName:808270b1-884c-43ed-ab2e-4f9ab3102332 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:53.804622451 +0000 UTC m=+36.172132477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls") pod "image-registry-846d7479f5-jpdkd" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332") : secret "image-registry-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804679 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804703 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls podName:d7e477da-8762-4d12-b6ad-7cfceec3dd1f nodeName:}" failed. No retries permitted until 2026-07-21 14:17:53.804679499 +0000 UTC m=+36.172189509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls") pod "image-registry-fcf8cfd48-sjz45" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f") : secret "image-registry-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804731 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert podName:bf728902-c923-4e7a-aeb1-94338a8b1d95 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:53.804720085 +0000 UTC m=+36.172230091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jvz68" (UID: "bf728902-c923-4e7a-aeb1-94338a8b1d95") : secret "networking-console-plugin-cert" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:52.804769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804891 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 14:17:52.805269 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:52.804942 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls podName:e444c9d9-48ba-4131-9b16-2c8e26296df0 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:53.80492761 +0000 UTC m=+36.172437618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls") pod "dns-default-76pfl" (UID: "e444c9d9-48ba-4131-9b16-2c8e26296df0") : secret "dns-default-metrics-tls" not found Jul 21 14:17:53.135036 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.134945 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:17:53.135175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.134945 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:17:53.135175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.134952 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:17:53.139201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.139048 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 14:17:53.139201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.139073 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 14:17:53.139201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.139073 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qcck5\"" Jul 21 14:17:53.139201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.139074 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2kbnq\"" Jul 21 14:17:53.324758 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.324698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" event={"ID":"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6","Type":"ContainerStarted","Data":"6f06cd34fdf52a83c2bc8c14babcefa8a116c2f877fd80016984079d96dc211c"} Jul 21 14:17:53.326272 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.326208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" event={"ID":"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1","Type":"ContainerStarted","Data":"5d0a4ecca3232f158076d2d0b0a425978e1359da9261283f62e0b1854c50329a"} Jul 21 14:17:53.327462 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.327441 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xp7xl" event={"ID":"d9f27890-994f-491e-b14e-52319993fc85","Type":"ContainerStarted","Data":"d43581a3fba38c5a54743c95b46babc2ceb92fc2ef7da58e82caca4ee082d0c5"} Jul 21 14:17:53.328801 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.328757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" event={"ID":"2d60d6e8-4ce0-43ce-a0c2-0ced19251643","Type":"ContainerStarted","Data":"d75f7fe529353d270c568d7515bf50daff6d5da738c72f6bcacccc7ebc4fd259"} Jul 21 14:17:53.330035 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.329978 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" event={"ID":"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2","Type":"ContainerStarted","Data":"2e34411014f7f72706bdbb40fc523468a535837b158c92417180b2f7038b6e1f"} Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.813616 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.813676 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.813790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.813807 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.813821 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.813831 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fcf8cfd48-sjz45: secret "image-registry-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.813892 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls podName:d7e477da-8762-4d12-b6ad-7cfceec3dd1f nodeName:}" failed. No retries permitted until 2026-07-21 14:17:55.813873082 +0000 UTC m=+38.181383102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls") pod "image-registry-fcf8cfd48-sjz45" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f") : secret "image-registry-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.813947 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.814050 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls podName:e444c9d9-48ba-4131-9b16-2c8e26296df0 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:55.814032508 +0000 UTC m=+38.181542531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls") pod "dns-default-76pfl" (UID: "e444c9d9-48ba-4131-9b16-2c8e26296df0") : secret "dns-default-metrics-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.814124 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.814135 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d7479f5-jpdkd: secret "image-registry-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.814168 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls podName:808270b1-884c-43ed-ab2e-4f9ab3102332 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:55.814157044 +0000 UTC m=+38.181667070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls") pod "image-registry-846d7479f5-jpdkd" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332") : secret "image-registry-tls" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.814221 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 14:17:53.814300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:53.814278 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert podName:bf728902-c923-4e7a-aeb1-94338a8b1d95 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:55.814267547 +0000 UTC m=+38.181777557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jvz68" (UID: "bf728902-c923-4e7a-aeb1-94338a8b1d95") : secret "networking-console-plugin-cert" not found Jul 21 14:17:53.853459 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:53.853401 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:54.150390 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:54.150302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:54.251294 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:54.251202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:54.616626 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:54.616548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:54.718929 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:54.718814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:55.031389 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.031336 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:55.121813 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.121784 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:55.345055 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.344956 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:55.460858 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.460820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:55.798741 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.798700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:55.833617 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.833587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:55.833753 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.833674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:55.833753 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.833712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:55.833855 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833756 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 14:17:55.833855 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833820 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls podName:e444c9d9-48ba-4131-9b16-2c8e26296df0 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:59.833801861 +0000 UTC m=+42.201311868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls") pod "dns-default-76pfl" (UID: "e444c9d9-48ba-4131-9b16-2c8e26296df0") : secret "dns-default-metrics-tls" not found Jul 21 14:17:55.833855 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833822 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:55.833855 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833838 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d7479f5-jpdkd: secret "image-registry-tls" not found Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833835 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833897 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fcf8cfd48-sjz45: secret "image-registry-tls" not found Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.833897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833931 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls podName:d7e477da-8762-4d12-b6ad-7cfceec3dd1f nodeName:}" failed. No retries permitted until 2026-07-21 14:17:59.833915188 +0000 UTC m=+42.201425200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls") pod "image-registry-fcf8cfd48-sjz45" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f") : secret "image-registry-tls" not found Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833952 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls podName:808270b1-884c-43ed-ab2e-4f9ab3102332 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:59.833939908 +0000 UTC m=+42.201449917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls") pod "image-registry-846d7479f5-jpdkd" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332") : secret "image-registry-tls" not found Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.833976 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 14:17:55.834060 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:55.834057 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert podName:bf728902-c923-4e7a-aeb1-94338a8b1d95 nodeName:}" failed. No retries permitted until 2026-07-21 14:17:59.834042352 +0000 UTC m=+42.201552378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jvz68" (UID: "bf728902-c923-4e7a-aeb1-94338a8b1d95") : secret "networking-console-plugin-cert" not found Jul 21 14:17:55.899562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:55.899525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:56.295306 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:56.295262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:56.402737 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:56.402651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:56.730615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:56.730579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:57.014136 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:57.014073 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:57.137656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:57.137614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:57.396418 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:57.396369 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:57.497892 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:57.497859 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:58.505075 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:58.505041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:58.606197 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:58.606160 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:17:59.867606 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:59.867566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:59.867617 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:59.867688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:17:59.867716 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867754 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867771 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fcf8cfd48-sjz45: secret "image-registry-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867811 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867827 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867851 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d7479f5-jpdkd: secret "image-registry-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867832 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls podName:d7e477da-8762-4d12-b6ad-7cfceec3dd1f nodeName:}" failed. No retries permitted until 2026-07-21 14:18:07.867813469 +0000 UTC m=+50.235323495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls") pod "image-registry-fcf8cfd48-sjz45" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f") : secret "image-registry-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867833 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867935 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls podName:e444c9d9-48ba-4131-9b16-2c8e26296df0 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:07.867915297 +0000 UTC m=+50.235425309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls") pod "dns-default-76pfl" (UID: "e444c9d9-48ba-4131-9b16-2c8e26296df0") : secret "dns-default-metrics-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867953 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls podName:808270b1-884c-43ed-ab2e-4f9ab3102332 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:07.867943859 +0000 UTC m=+50.235453865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls") pod "image-registry-846d7479f5-jpdkd" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332") : secret "image-registry-tls" not found Jul 21 14:17:59.868062 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:17:59.867971 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert podName:bf728902-c923-4e7a-aeb1-94338a8b1d95 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:07.86796242 +0000 UTC m=+50.235472428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jvz68" (UID: "bf728902-c923-4e7a-aeb1-94338a8b1d95") : secret "networking-console-plugin-cert" not found Jul 21 14:18:01.352726 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.352632 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" event={"ID":"2d60d6e8-4ce0-43ce-a0c2-0ced19251643","Type":"ContainerStarted","Data":"60599aa24eefea9f9e2b9baa397da22a15a769474bd374e3036c531ea68d8c7f"} Jul 21 14:18:01.355993 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.355963 2570 generic.go:356] "Generic (PLEG): container finished" podID="25040ae5-5c9b-42fd-9271-d2b3e472162d" containerID="62eaf55fe53d123cae307362ee138e565f8eddcf5ccf6d3313ea2b99cefb1dbc" exitCode=0 Jul 21 14:18:01.356121 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.356038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerDied","Data":"62eaf55fe53d123cae307362ee138e565f8eddcf5ccf6d3313ea2b99cefb1dbc"} Jul 21 14:18:01.358019 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.357982 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" event={"ID":"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2","Type":"ContainerStarted","Data":"51051325d964c1aec7e498a7d34b7fb0b1d2a7f4a18922cdf30b6e1571c78910"} Jul 21 14:18:01.359602 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.359582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" event={"ID":"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6","Type":"ContainerStarted","Data":"75e1f3cd23c4d9dbc3d285fb5cdeb7fbc162981009ef1cc791a6a7c022a669c2"} Jul 21 14:18:01.361210 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.361196 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/0.log" Jul 21 14:18:01.361305 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.361227 2570 generic.go:356] "Generic (PLEG): container finished" podID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" containerID="03cccff2305a331c0c1a071c1ca41b5b9142cfe24de53932aeb0c8675eb8d024" exitCode=255 Jul 21 14:18:01.361359 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.361317 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" event={"ID":"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1","Type":"ContainerDied","Data":"03cccff2305a331c0c1a071c1ca41b5b9142cfe24de53932aeb0c8675eb8d024"} Jul 21 14:18:01.361569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.361458 2570 scope.go:117] "RemoveContainer" containerID="03cccff2305a331c0c1a071c1ca41b5b9142cfe24de53932aeb0c8675eb8d024" Jul 21 14:18:01.365226 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.364420 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xp7xl" event={"ID":"d9f27890-994f-491e-b14e-52319993fc85","Type":"ContainerStarted","Data":"4177ddb107cf76baf42eb21b50c9b62fca801802c72b9dc718ac2ebf94239d90"} Jul 21 14:18:01.373787 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.373745 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-hqml7" podStartSLOduration=30.1327981 podStartE2EDuration="38.373729402s" totalImagesPullingTime="8.240931302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:01.372169728 +0000 UTC m=+43.739679757" watchObservedRunningTime="2026-07-21 14:18:01.373729402 +0000 UTC m=+43.741239410" Jul 21 14:18:01.422986 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.418709 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" podStartSLOduration=31.167532044 podStartE2EDuration="39.418684456s" totalImagesPullingTime="8.251152412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:01.395592231 +0000 UTC m=+43.763102260" watchObservedRunningTime="2026-07-21 14:18:01.418684456 +0000 UTC m=+43.786194485" Jul 21 14:18:01.449049 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:01.448987 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" podStartSLOduration=31.174198772 podStartE2EDuration="39.448973775s" totalImagesPullingTime="8.274775003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:01.44854224 +0000 UTC m=+43.816052268" watchObservedRunningTime="2026-07-21 14:18:01.448973775 +0000 UTC m=+43.816483802" Jul 21 14:18:02.352744 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.352704 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:02.355525 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.355499 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:02.355638 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.355562 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:02.373059 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.373034 2570 generic.go:356] "Generic (PLEG): container finished" podID="25040ae5-5c9b-42fd-9271-d2b3e472162d" containerID="e31240053da00b9eac14ce2bd71c62f7a6ba648ef0d6437b40235dc196df8598" exitCode=0 Jul 21 14:18:02.373200 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.373114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerDied","Data":"e31240053da00b9eac14ce2bd71c62f7a6ba648ef0d6437b40235dc196df8598"} Jul 21 14:18:02.374740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.374723 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/1.log" Jul 21 14:18:02.375103 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.375089 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/0.log" Jul 21 14:18:02.375171 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.375122 2570 generic.go:356] "Generic (PLEG): container finished" podID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" containerID="e0bb70a0bab06d7d71da1fe93bfdf561d9381daf0ff9d98da7f855076d56e2e3" exitCode=255 Jul 21 14:18:02.375231 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.375215 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" event={"ID":"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1","Type":"ContainerDied","Data":"e0bb70a0bab06d7d71da1fe93bfdf561d9381daf0ff9d98da7f855076d56e2e3"} Jul 21 14:18:02.375284 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.375261 2570 scope.go:117] "RemoveContainer" containerID="03cccff2305a331c0c1a071c1ca41b5b9142cfe24de53932aeb0c8675eb8d024" Jul 21 14:18:02.375471 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.375440 2570 scope.go:117] "RemoveContainer" containerID="e0bb70a0bab06d7d71da1fe93bfdf561d9381daf0ff9d98da7f855076d56e2e3" Jul 21 14:18:02.375679 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:02.375642 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-kg4q9_openshift-console-operator(f7f3c68c-9553-4b61-a33f-c30d4e11f9c1)\"" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podUID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" Jul 21 14:18:02.408763 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.408710 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xp7xl" podStartSLOduration=2.185650663 podStartE2EDuration="10.408688464s" totalImagesPullingTime="8.223037801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:01.510903285 +0000 UTC m=+43.878413315" watchObservedRunningTime="2026-07-21 14:18:02.408688464 +0000 UTC m=+44.776198493" Jul 21 14:18:02.482266 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:02.482234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:03.205769 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.205734 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:03.326257 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.326223 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls"] Jul 21 14:18:03.329556 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.329524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:03.349344 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.349319 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls"] Jul 21 14:18:03.349456 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.349414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" Jul 21 14:18:03.354219 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.354189 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 21 14:18:03.354824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.354426 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 21 14:18:03.366492 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.366466 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-njrxz\"" Jul 21 14:18:03.378963 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.378940 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/1.log" Jul 21 14:18:03.379349 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.379324 2570 scope.go:117] "RemoveContainer" containerID="e0bb70a0bab06d7d71da1fe93bfdf561d9381daf0ff9d98da7f855076d56e2e3" Jul 21 14:18:03.379521 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:03.379501 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-kg4q9_openshift-console-operator(f7f3c68c-9553-4b61-a33f-c30d4e11f9c1)\"" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podUID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" Jul 21 14:18:03.382225 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.382194 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" event={"ID":"25040ae5-5c9b-42fd-9271-d2b3e472162d","Type":"ContainerStarted","Data":"243a392550979e9f031cb34a3ae393c134d6b8a26833a6c9bff3a560ad3b91df"} Jul 21 14:18:03.403410 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.403369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b7d\" (UniqueName: \"kubernetes.io/projected/4302d816-607c-4944-92ed-8618212a8ecb-kube-api-access-h6b7d\") pod \"migrator-7cc9c7ffd7-4fkls\" (UID: \"4302d816-607c-4944-92ed-8618212a8ecb\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" Jul 21 14:18:03.423281 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.423254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:03.431514 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.431477 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xmsqf" podStartSLOduration=8.789955234 podStartE2EDuration="45.431463154s" totalImagesPullingTime="36.64150792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:03.429748585 +0000 UTC m=+45.797258612" watchObservedRunningTime="2026-07-21 14:18:03.431463154 +0000 UTC m=+45.798973182" Jul 21 14:18:03.504659 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.504581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b7d\" (UniqueName: \"kubernetes.io/projected/4302d816-607c-4944-92ed-8618212a8ecb-kube-api-access-h6b7d\") pod \"migrator-7cc9c7ffd7-4fkls\" (UID: \"4302d816-607c-4944-92ed-8618212a8ecb\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" Jul 21 14:18:03.516143 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.516115 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b7d\" (UniqueName: \"kubernetes.io/projected/4302d816-607c-4944-92ed-8618212a8ecb-kube-api-access-h6b7d\") pod \"migrator-7cc9c7ffd7-4fkls\" (UID: \"4302d816-607c-4944-92ed-8618212a8ecb\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" Jul 21 14:18:03.528550 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.528521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:03.660569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.660537 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" Jul 21 14:18:03.669609 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.669577 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:03.796294 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:03.796223 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls"] Jul 21 14:18:03.800858 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:03.800833 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4302d816_607c_4944_92ed_8618212a8ecb.slice/crio-abcd94932b70680658f61eb9cb74feb8100e2f02fe1a997e010c5e511b26fc20 WatchSource:0}: Error finding container abcd94932b70680658f61eb9cb74feb8100e2f02fe1a997e010c5e511b26fc20: Status 404 returned error can't find the container with id abcd94932b70680658f61eb9cb74feb8100e2f02fe1a997e010c5e511b26fc20 Jul 21 14:18:04.330976 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:04.330942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:04.385883 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:04.385851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" event={"ID":"4302d816-607c-4944-92ed-8618212a8ecb","Type":"ContainerStarted","Data":"abcd94932b70680658f61eb9cb74feb8100e2f02fe1a997e010c5e511b26fc20"} Jul 21 14:18:04.404222 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:04.404198 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:05.450644 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.450611 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-62znd"] Jul 21 14:18:05.470091 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.470064 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-62znd"] Jul 21 14:18:05.470222 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.470176 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.472930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.472896 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 21 14:18:05.472930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.472918 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 21 14:18:05.474950 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.474930 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 21 14:18:05.474950 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.474936 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c8kh6\"" Jul 21 14:18:05.475098 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.474988 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 21 14:18:05.501996 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.501973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:05.521156 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.521133 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01597742-1793-424f-803d-5227453c02a2-signing-key\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.521258 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.521165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2947\" (UniqueName: \"kubernetes.io/projected/01597742-1793-424f-803d-5227453c02a2-kube-api-access-v2947\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.521258 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.521202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01597742-1793-424f-803d-5227453c02a2-signing-cabundle\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.581542 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.581520 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:05.621877 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.621849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01597742-1793-424f-803d-5227453c02a2-signing-key\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.621877 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.621879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2947\" (UniqueName: \"kubernetes.io/projected/01597742-1793-424f-803d-5227453c02a2-kube-api-access-v2947\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.622093 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.621906 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01597742-1793-424f-803d-5227453c02a2-signing-cabundle\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.622537 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.622518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/01597742-1793-424f-803d-5227453c02a2-signing-cabundle\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.624067 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.624049 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/01597742-1793-424f-803d-5227453c02a2-signing-key\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.634548 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.634530 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2947\" (UniqueName: \"kubernetes.io/projected/01597742-1793-424f-803d-5227453c02a2-kube-api-access-v2947\") pod \"service-ca-58598695f7-62znd\" (UID: \"01597742-1793-424f-803d-5227453c02a2\") " pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.779455 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.779435 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-62znd" Jul 21 14:18:05.918104 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:05.918077 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-62znd"] Jul 21 14:18:05.920970 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:05.920945 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01597742_1793_424f_803d_5227453c02a2.slice/crio-821c8698316db432223db83b427b3c25d0a2e8df3080667eba8c3687e2d74766 WatchSource:0}: Error finding container 821c8698316db432223db83b427b3c25d0a2e8df3080667eba8c3687e2d74766: Status 404 returned error can't find the container with id 821c8698316db432223db83b427b3c25d0a2e8df3080667eba8c3687e2d74766 Jul 21 14:18:06.393352 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.393299 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" event={"ID":"4302d816-607c-4944-92ed-8618212a8ecb","Type":"ContainerStarted","Data":"6e109c5f954ed1f2257d3e0b13de5fbf2acc364130737b934e4c487415c9d597"} Jul 21 14:18:06.393541 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.393370 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" event={"ID":"4302d816-607c-4944-92ed-8618212a8ecb","Type":"ContainerStarted","Data":"48c048adb2d86c59f2460f64faa1f7432b02cb8221a80568879e92de89726890"} Jul 21 14:18:06.394940 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.394913 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-62znd" event={"ID":"01597742-1793-424f-803d-5227453c02a2","Type":"ContainerStarted","Data":"fa5104c60df0655091e2a95258b47d27d90c608874281fe6b3e740b7a5cba4b4"} Jul 21 14:18:06.395070 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.394959 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-62znd" event={"ID":"01597742-1793-424f-803d-5227453c02a2","Type":"ContainerStarted","Data":"821c8698316db432223db83b427b3c25d0a2e8df3080667eba8c3687e2d74766"} Jul 21 14:18:06.433850 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.433809 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-4fkls" podStartSLOduration=1.706103602 podStartE2EDuration="3.433794421s" totalImagesPullingTime="1.727690819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:06.413821577 +0000 UTC m=+48.781331603" watchObservedRunningTime="2026-07-21 14:18:06.433794421 +0000 UTC m=+48.801304448" Jul 21 14:18:06.434317 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.434290 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-62znd" podStartSLOduration=1.434281401 podStartE2EDuration="1.434281401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:06.433234683 +0000 UTC m=+48.800744725" watchObservedRunningTime="2026-07-21 14:18:06.434281401 +0000 UTC m=+48.801791428" Jul 21 14:18:06.523096 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:06.523060 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:07.941869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:07.941825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:07.941877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:07.941948 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:07.941986 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.941983 2570 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942081 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert podName:bf728902-c923-4e7a-aeb1-94338a8b1d95 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.942057977 +0000 UTC m=+66.309567984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-jvz68" (UID: "bf728902-c923-4e7a-aeb1-94338a8b1d95") : secret "networking-console-plugin-cert" not found Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942135 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942149 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846d7479f5-jpdkd: secret "image-registry-tls" not found Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942194 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls podName:808270b1-884c-43ed-ab2e-4f9ab3102332 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.942179977 +0000 UTC m=+66.309690001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls") pod "image-registry-846d7479f5-jpdkd" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332") : secret "image-registry-tls" not found Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942256 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 14:18:07.942320 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942283 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls podName:e444c9d9-48ba-4131-9b16-2c8e26296df0 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.9422727 +0000 UTC m=+66.309782711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls") pod "dns-default-76pfl" (UID: "e444c9d9-48ba-4131-9b16-2c8e26296df0") : secret "dns-default-metrics-tls" not found Jul 21 14:18:07.942669 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942337 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 14:18:07.942669 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942346 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fcf8cfd48-sjz45: secret "image-registry-tls" not found Jul 21 14:18:07.942669 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:07.942372 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls podName:d7e477da-8762-4d12-b6ad-7cfceec3dd1f nodeName:}" failed. No retries permitted until 2026-07-21 14:18:23.942362474 +0000 UTC m=+66.309872486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls") pod "image-registry-fcf8cfd48-sjz45" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f") : secret "image-registry-tls" not found Jul 21 14:18:08.070315 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:08.070281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:08.179050 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:08.178986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:08.624598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:08.624566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:09.221278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:09.221230 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:09.341919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:09.341879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:09.628775 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:09.628687 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:10.029226 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:10.029197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:10.226070 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:10.226036 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:10.836123 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:10.836092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:11.342754 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:11.342714 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:11.445350 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:11.445318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:11.545784 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:11.545755 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:11.680922 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:11.680890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:12.028709 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.028618 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:12.245419 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.245385 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:12.355077 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.354969 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:12.355077 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.355056 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:12.355469 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.355382 2570 scope.go:117] "RemoveContainer" containerID="e0bb70a0bab06d7d71da1fe93bfdf561d9381daf0ff9d98da7f855076d56e2e3" Jul 21 14:18:12.414153 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.414134 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/1.log" Jul 21 14:18:12.414273 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.414196 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" event={"ID":"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1","Type":"ContainerStarted","Data":"6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764"} Jul 21 14:18:12.414475 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.414443 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:12.415641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.415606 2570 patch_prober.go:28] interesting pod/console-operator-565b7b569d-kg4q9 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body= Jul 21 14:18:12.415713 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.415644 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podUID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" Jul 21 14:18:12.441531 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.441478 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podStartSLOduration=46.157546439 podStartE2EDuration="54.441465592s" totalImagesPullingTime="8.283919153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:12.441165508 +0000 UTC m=+54.808675536" watchObservedRunningTime="2026-07-21 14:18:12.441465592 +0000 UTC m=+54.808975620" Jul 21 14:18:12.797603 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.797567 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:12.876343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:12.876304 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:13.419300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.419273 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:18:13.419714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.419661 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/1.log" Jul 21 14:18:13.419714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.419688 2570 generic.go:356] "Generic (PLEG): container finished" podID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" containerID="6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764" exitCode=255 Jul 21 14:18:13.419782 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.419732 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" event={"ID":"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1","Type":"ContainerDied","Data":"6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764"} Jul 21 14:18:13.419782 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.419771 2570 scope.go:117] "RemoveContainer" containerID="e0bb70a0bab06d7d71da1fe93bfdf561d9381daf0ff9d98da7f855076d56e2e3" Jul 21 14:18:13.419995 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.419978 2570 scope.go:117] "RemoveContainer" containerID="6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764" Jul 21 14:18:13.420226 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:13.420203 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-kg4q9_openshift-console-operator(f7f3c68c-9553-4b61-a33f-c30d4e11f9c1)\"" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podUID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" Jul 21 14:18:13.444122 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.444095 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:13.623879 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.623843 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:13.822188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:13.822068 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:14.029676 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:14.029617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:14.237391 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:14.237362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:14.423578 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:14.423541 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:18:14.423994 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:14.423840 2570 scope.go:117] "RemoveContainer" containerID="6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764" Jul 21 14:18:14.423994 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:14.423976 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-kg4q9_openshift-console-operator(f7f3c68c-9553-4b61-a33f-c30d4e11f9c1)\"" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podUID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" Jul 21 14:18:15.039408 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:15.039369 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:15.424358 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:15.424324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:15.631538 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:15.631504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:15.838854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:15.838758 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:19.099784 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:19.099751 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:19.177031 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:19.176982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:19.320919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:19.320895 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wbx42" Jul 21 14:18:22.355061 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:22.355002 2570 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:22.355438 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:22.355421 2570 scope.go:117] "RemoveContainer" containerID="6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764" Jul 21 14:18:22.355597 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:22.355575 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-kg4q9_openshift-console-operator(f7f3c68c-9553-4b61-a33f-c30d4e11f9c1)\"" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" podUID="f7f3c68c-9553-4b61-a33f-c30d4e11f9c1" Jul 21 14:18:23.871229 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.871194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:18:23.871658 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.871306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:18:23.874124 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.874104 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 14:18:23.874274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.874258 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 14:18:23.884465 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.884441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1f9a0b12-10a7-4628-92ed-9d9651940451-original-pull-secret\") pod \"global-pull-secret-syncer-p447f\" (UID: \"1f9a0b12-10a7-4628-92ed-9d9651940451\") " pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:18:23.884610 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.884590 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0730acd0-6273-431f-a01f-b5eefe741362-metrics-certs\") pod \"network-metrics-daemon-rdkx6\" (UID: \"0730acd0-6273-431f-a01f-b5eefe741362\") " pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:18:23.972362 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.972330 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:18:23.972508 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.972386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:18:23.972508 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.972411 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:18:23.972508 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.972468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:23.974540 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.974515 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e444c9d9-48ba-4131-9b16-2c8e26296df0-metrics-tls\") pod \"dns-default-76pfl\" (UID: \"e444c9d9-48ba-4131-9b16-2c8e26296df0\") " pod="openshift-dns/dns-default-76pfl" Jul 21 14:18:23.974642 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.974620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"image-registry-846d7479f5-jpdkd\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:18:23.974766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.974746 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"image-registry-fcf8cfd48-sjz45\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:23.974801 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:23.974783 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf728902-c923-4e7a-aeb1-94338a8b1d95-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-jvz68\" (UID: \"bf728902-c923-4e7a-aeb1-94338a8b1d95\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:18:24.047360 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.047328 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p447f" Jul 21 14:18:24.062312 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.062292 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2kbnq\"" Jul 21 14:18:24.069687 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.069667 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rdkx6" Jul 21 14:18:24.073193 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.073172 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:18:24.075475 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.075453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfj9\" (UniqueName: \"kubernetes.io/projected/1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e-kube-api-access-2lfj9\") pod \"network-check-target-mr7zj\" (UID: \"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e\") " pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:18:24.177130 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.177102 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2s4f2\"" Jul 21 14:18:24.177590 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.177568 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p447f"] Jul 21 14:18:24.179918 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.179893 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9a0b12_10a7_4628_92ed_9d9651940451.slice/crio-debc2f670ee8b02cdf09502ece69ccebda4eb6edb1149110b78f20838d56bee3 WatchSource:0}: Error finding container debc2f670ee8b02cdf09502ece69ccebda4eb6edb1149110b78f20838d56bee3: Status 404 returned error can't find the container with id debc2f670ee8b02cdf09502ece69ccebda4eb6edb1149110b78f20838d56bee3 Jul 21 14:18:24.185318 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.185299 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:18:24.190852 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.190829 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:24.196404 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.196383 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rdkx6"] Jul 21 14:18:24.199729 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.199693 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0730acd0_6273_431f_a01f_b5eefe741362.slice/crio-3da3af6aa0108d64285cd9ce5a96420a3b26d377a0f032d6fee59c0097f3d2bd WatchSource:0}: Error finding container 3da3af6aa0108d64285cd9ce5a96420a3b26d377a0f032d6fee59c0097f3d2bd: Status 404 returned error can't find the container with id 3da3af6aa0108d64285cd9ce5a96420a3b26d377a0f032d6fee59c0097f3d2bd Jul 21 14:18:24.208594 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.208571 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-87nvz\"" Jul 21 14:18:24.216161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.216143 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" Jul 21 14:18:24.216161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.216158 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rkkbj\"" Jul 21 14:18:24.224396 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.224358 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-76pfl" Jul 21 14:18:24.351054 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.350837 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846d7479f5-jpdkd"] Jul 21 14:18:24.352443 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.352414 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808270b1_884c_43ed_ab2e_4f9ab3102332.slice/crio-4dc0e6c852282bcc0c9bdd66cecf082acaefdb8afbdd4c1ded4a332c5d0a9588 WatchSource:0}: Error finding container 4dc0e6c852282bcc0c9bdd66cecf082acaefdb8afbdd4c1ded4a332c5d0a9588: Status 404 returned error can't find the container with id 4dc0e6c852282bcc0c9bdd66cecf082acaefdb8afbdd4c1ded4a332c5d0a9588 Jul 21 14:18:24.356312 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.356287 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qcck5\"" Jul 21 14:18:24.361145 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.361078 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fcf8cfd48-sjz45"] Jul 21 14:18:24.363505 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.363485 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:18:24.365140 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.365115 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e477da_8762_4d12_b6ad_7cfceec3dd1f.slice/crio-1bd40a64625ec926d8ea92721992a308299827cff01323db7bccd1da116953db WatchSource:0}: Error finding container 1bd40a64625ec926d8ea92721992a308299827cff01323db7bccd1da116953db: Status 404 returned error can't find the container with id 1bd40a64625ec926d8ea92721992a308299827cff01323db7bccd1da116953db Jul 21 14:18:24.388546 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.388519 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-76pfl"] Jul 21 14:18:24.391690 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.391653 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode444c9d9_48ba_4131_9b16_2c8e26296df0.slice/crio-9f2a1632dc8c07608ce44d5e66eed82907e4d0cc10ab014e51864f91498cc7d3 WatchSource:0}: Error finding container 9f2a1632dc8c07608ce44d5e66eed82907e4d0cc10ab014e51864f91498cc7d3: Status 404 returned error can't find the container with id 9f2a1632dc8c07608ce44d5e66eed82907e4d0cc10ab014e51864f91498cc7d3 Jul 21 14:18:24.402146 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.402078 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68"] Jul 21 14:18:24.405438 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.405411 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf728902_c923_4e7a_aeb1_94338a8b1d95.slice/crio-6e14a695397e3e4356c0f1e616adfee43d93f23a55dbec0978913946b8b0f04b WatchSource:0}: Error finding container 6e14a695397e3e4356c0f1e616adfee43d93f23a55dbec0978913946b8b0f04b: Status 404 returned error can't find the container with id 6e14a695397e3e4356c0f1e616adfee43d93f23a55dbec0978913946b8b0f04b Jul 21 14:18:24.455584 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.455529 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" event={"ID":"bf728902-c923-4e7a-aeb1-94338a8b1d95","Type":"ContainerStarted","Data":"6e14a695397e3e4356c0f1e616adfee43d93f23a55dbec0978913946b8b0f04b"} Jul 21 14:18:24.457453 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.457388 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" event={"ID":"d7e477da-8762-4d12-b6ad-7cfceec3dd1f","Type":"ContainerStarted","Data":"8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480"} Jul 21 14:18:24.457453 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.457430 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" event={"ID":"d7e477da-8762-4d12-b6ad-7cfceec3dd1f","Type":"ContainerStarted","Data":"1bd40a64625ec926d8ea92721992a308299827cff01323db7bccd1da116953db"} Jul 21 14:18:24.457628 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.457470 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:24.458932 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.458903 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" event={"ID":"808270b1-884c-43ed-ab2e-4f9ab3102332","Type":"ContainerStarted","Data":"f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d"} Jul 21 14:18:24.459067 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.458943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" event={"ID":"808270b1-884c-43ed-ab2e-4f9ab3102332","Type":"ContainerStarted","Data":"4dc0e6c852282bcc0c9bdd66cecf082acaefdb8afbdd4c1ded4a332c5d0a9588"} Jul 21 14:18:24.459067 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.458968 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:18:24.460229 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.460206 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdkx6" event={"ID":"0730acd0-6273-431f-a01f-b5eefe741362","Type":"ContainerStarted","Data":"3da3af6aa0108d64285cd9ce5a96420a3b26d377a0f032d6fee59c0097f3d2bd"} Jul 21 14:18:24.461332 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.461313 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p447f" event={"ID":"1f9a0b12-10a7-4628-92ed-9d9651940451","Type":"ContainerStarted","Data":"debc2f670ee8b02cdf09502ece69ccebda4eb6edb1149110b78f20838d56bee3"} Jul 21 14:18:24.462471 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.462449 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76pfl" event={"ID":"e444c9d9-48ba-4131-9b16-2c8e26296df0","Type":"ContainerStarted","Data":"9f2a1632dc8c07608ce44d5e66eed82907e4d0cc10ab014e51864f91498cc7d3"} Jul 21 14:18:24.480313 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.480268 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" podStartSLOduration=61.480253849 podStartE2EDuration="1m1.480253849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:24.478958869 +0000 UTC m=+66.846468899" watchObservedRunningTime="2026-07-21 14:18:24.480253849 +0000 UTC m=+66.847763876" Jul 21 14:18:24.503058 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.502975 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" podStartSLOduration=66.502955209 podStartE2EDuration="1m6.502955209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:24.502287332 +0000 UTC m=+66.869797360" watchObservedRunningTime="2026-07-21 14:18:24.502955209 +0000 UTC m=+66.870465241" Jul 21 14:18:24.505997 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:24.505970 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mr7zj"] Jul 21 14:18:24.509317 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:24.509291 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8b881a_3a2c_46b9_bc19_5ffd5bb00c7e.slice/crio-e87dcae0061291060252b5dae3048df02691884ebfcb10473fba5b7c60b0fcb0 WatchSource:0}: Error finding container e87dcae0061291060252b5dae3048df02691884ebfcb10473fba5b7c60b0fcb0: Status 404 returned error can't find the container with id e87dcae0061291060252b5dae3048df02691884ebfcb10473fba5b7c60b0fcb0 Jul 21 14:18:25.470369 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:25.470317 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdkx6" event={"ID":"0730acd0-6273-431f-a01f-b5eefe741362","Type":"ContainerStarted","Data":"bea9f9c61293900b08725f0f476fddb222f611e041bd727e06628b16b1a24e89"} Jul 21 14:18:25.475897 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:25.475402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mr7zj" event={"ID":"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e","Type":"ContainerStarted","Data":"67423cbe355e1f341947edb8baa1f6e07fdad8d29ee31182362d91ed78b10bed"} Jul 21 14:18:25.475897 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:25.475449 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mr7zj" event={"ID":"1a8b881a-3a2c-46b9-bc19-5ffd5bb00c7e","Type":"ContainerStarted","Data":"e87dcae0061291060252b5dae3048df02691884ebfcb10473fba5b7c60b0fcb0"} Jul 21 14:18:25.475897 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:25.475694 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:18:26.480327 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.480288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76pfl" event={"ID":"e444c9d9-48ba-4131-9b16-2c8e26296df0","Type":"ContainerStarted","Data":"a2d6433ef5ed6893f667670a71bcfa1359ad3e5d71f25f918ca90226c017d716"} Jul 21 14:18:26.480717 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.480352 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-76pfl" event={"ID":"e444c9d9-48ba-4131-9b16-2c8e26296df0","Type":"ContainerStarted","Data":"78921016af64ad1c01a297749fcd50e254243598da3a52b9ff8626a216313c90"} Jul 21 14:18:26.480717 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.480383 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-76pfl" Jul 21 14:18:26.482350 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.482326 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rdkx6" event={"ID":"0730acd0-6273-431f-a01f-b5eefe741362","Type":"ContainerStarted","Data":"975b8036b23356a1f28f76256e223706bf90a09cbb165576887c0c95fe5a231e"} Jul 21 14:18:26.499751 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.499673 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mr7zj" podStartSLOduration=68.499655892 podStartE2EDuration="1m8.499655892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:25.493727441 +0000 UTC m=+67.861237470" watchObservedRunningTime="2026-07-21 14:18:26.499655892 +0000 UTC m=+68.867165921" Jul 21 14:18:26.501569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.500336 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-76pfl" podStartSLOduration=32.648200598 podStartE2EDuration="34.500318796s" totalImagesPullingTime="1.852118198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:26.498442001 +0000 UTC m=+68.865952030" watchObservedRunningTime="2026-07-21 14:18:26.500318796 +0000 UTC m=+68.867828849" Jul 21 14:18:26.524889 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:26.524837 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rdkx6" podStartSLOduration=67.404287165 podStartE2EDuration="1m8.524819024s" totalImagesPullingTime="1.120531859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:26.524246793 +0000 UTC m=+68.891756824" watchObservedRunningTime="2026-07-21 14:18:26.524819024 +0000 UTC m=+68.892329050" Jul 21 14:18:28.888141 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.888099 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n"] Jul 21 14:18:28.890272 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.890205 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:28.894084 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.894058 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 14:18:28.894200 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.894093 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ckwqq\"" Jul 21 14:18:28.895326 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.895306 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 14:18:28.895426 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.895348 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 14:18:28.895426 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.895285 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 14:18:28.904645 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.904623 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n"] Jul 21 14:18:28.907136 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.907023 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws"] Jul 21 14:18:28.909363 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.909344 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:28.910638 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.910591 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d7bf988cb-t286m"] Jul 21 14:18:28.912098 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.912078 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j9z4n\"" Jul 21 14:18:28.912214 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.912197 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 14:18:28.912524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.912506 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 14:18:28.912733 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.912708 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:18:28.913086 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.913035 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg"] Jul 21 14:18:28.913222 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.913205 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:28.915171 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.915152 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n"] Jul 21 14:18:28.915264 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.915224 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:28.915912 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.915892 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 14:18:28.917913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.917270 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 14:18:28.917913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.917367 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 14:18:28.917913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.917370 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 14:18:28.917913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.917567 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:28.917913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.917847 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hv26g\"" Jul 21 14:18:28.918264 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.918172 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 14:18:28.918330 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.918311 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 14:18:28.918501 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.918476 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 21 14:18:28.918900 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.918836 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt"] Jul 21 14:18:28.920954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.920775 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 14:18:28.920954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.920841 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 14:18:28.920954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.920894 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 21 14:18:28.920954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.920896 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 14:18:28.921398 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.921076 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 14:18:28.921398 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.921118 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:28.921398 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.921128 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 21 14:18:28.921398 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.921319 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 21 14:18:28.923427 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.923405 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws"] Jul 21 14:18:28.927727 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.927706 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-njvdz\"" Jul 21 14:18:28.928053 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.928036 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 21 14:18:28.928318 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.928292 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d7bf988cb-t286m"] Jul 21 14:18:28.932884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.932838 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg"] Jul 21 14:18:28.934220 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.934201 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt"] Jul 21 14:18:28.941190 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.941168 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n"] Jul 21 14:18:28.969941 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:28.969918 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fcf8cfd48-sjz45"] Jul 21 14:18:29.013502 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mks87\" (UniqueName: \"kubernetes.io/projected/7eb76497-1a07-46fa-8b0d-796471c880b9-kube-api-access-mks87\") pod \"cluster-samples-operator-79b7c55f87-m97ws\" (UID: \"7eb76497-1a07-46fa-8b0d-796471c880b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.013613 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-96c798986-4m2pt\" (UID: \"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.013677 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5hh\" (UniqueName: \"kubernetes.io/projected/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-kube-api-access-vz5hh\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.013677 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013649 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8d4098-96bf-4828-adba-3a50ea03d86c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.013780 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013682 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.013780 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013740 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-ca\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.013780 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-metrics-certs\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.013927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013813 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d18d8dde-86bd-4fff-8a27-a0c0280720a4-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.013927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013838 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-hub\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.013927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.013927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnbr\" (UniqueName: \"kubernetes.io/projected/6e8d4098-96bf-4828-adba-3a50ea03d86c-kube-api-access-wjnbr\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.013927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.013914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50910e2-559c-4954-b35e-d877abde0ef4-service-ca-bundle\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.014185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014001 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d18d8dde-86bd-4fff-8a27-a0c0280720a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.014185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-stats-auth\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.014185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014081 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-tmp\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.014185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014104 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-default-certificate\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.014185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.014185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5qrt\" (UniqueName: \"kubernetes.io/projected/2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48-kube-api-access-h5qrt\") pod \"managed-serviceaccount-addon-agent-96c798986-4m2pt\" (UID: \"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.014441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014205 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7eb76497-1a07-46fa-8b0d-796471c880b9-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m97ws\" (UID: \"7eb76497-1a07-46fa-8b0d-796471c880b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.014441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014255 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfwm\" (UniqueName: \"kubernetes.io/projected/d50910e2-559c-4954-b35e-d877abde0ef4-kube-api-access-9pfwm\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.014441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.014290 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtp2\" (UniqueName: \"kubernetes.io/projected/d18d8dde-86bd-4fff-8a27-a0c0280720a4-kube-api-access-zbtp2\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.022230 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.022205 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz"] Jul 21 14:18:29.022729 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.022692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:29.024809 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.024792 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kglk5"] Jul 21 14:18:29.024929 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.024911 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" Jul 21 14:18:29.027062 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.027045 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.027732 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.027705 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 14:18:29.027821 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.027798 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dcbnn\"" Jul 21 14:18:29.027882 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.027798 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 14:18:29.029888 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.029866 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 14:18:29.030473 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.030456 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 14:18:29.030567 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.030503 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h7x6x\"" Jul 21 14:18:29.030665 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.030634 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 14:18:29.030731 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.030668 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 14:18:29.039857 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.039638 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz"] Jul 21 14:18:29.042268 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.042245 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 14:18:29.053874 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.053854 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kglk5"] Jul 21 14:18:29.115068 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115041 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.115203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-tmp\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.115203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-ca\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.115203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115118 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-metrics-certs\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.115203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d18d8dde-86bd-4fff-8a27-a0c0280720a4-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.115421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-hub\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.115421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-serving-cert\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.115421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115264 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.115421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnbr\" (UniqueName: \"kubernetes.io/projected/6e8d4098-96bf-4828-adba-3a50ea03d86c-kube-api-access-wjnbr\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.115421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50910e2-559c-4954-b35e-d877abde0ef4-service-ca-bundle\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.115421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-snapshots\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d18d8dde-86bd-4fff-8a27-a0c0280720a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115454 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-stats-auth\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115479 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgw5l\" (UniqueName: \"kubernetes.io/projected/5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd-kube-api-access-xgw5l\") pod \"volume-data-source-validator-78cf4c6f6-pq8nz\" (UID: \"5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-tmp\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115536 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-default-certificate\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115573 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115611 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115648 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5qrt\" (UniqueName: \"kubernetes.io/projected/2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48-kube-api-access-h5qrt\") pod \"managed-serviceaccount-addon-agent-96c798986-4m2pt\" (UID: \"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115675 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7eb76497-1a07-46fa-8b0d-796471c880b9-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m97ws\" (UID: \"7eb76497-1a07-46fa-8b0d-796471c880b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.115715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfwm\" (UniqueName: \"kubernetes.io/projected/d50910e2-559c-4954-b35e-d877abde0ef4-kube-api-access-9pfwm\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115731 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtp2\" (UniqueName: \"kubernetes.io/projected/d18d8dde-86bd-4fff-8a27-a0c0280720a4-kube-api-access-zbtp2\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115765 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-service-ca-bundle\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fklzd\" (UniqueName: \"kubernetes.io/projected/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-kube-api-access-fklzd\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115827 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mks87\" (UniqueName: \"kubernetes.io/projected/7eb76497-1a07-46fa-8b0d-796471c880b9-kube-api-access-mks87\") pod \"cluster-samples-operator-79b7c55f87-m97ws\" (UID: \"7eb76497-1a07-46fa-8b0d-796471c880b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115856 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-96c798986-4m2pt\" (UID: \"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115879 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5hh\" (UniqueName: \"kubernetes.io/projected/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-kube-api-access-vz5hh\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.115907 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8d4098-96bf-4828-adba-3a50ea03d86c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.116217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.116073 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d50910e2-559c-4954-b35e-d877abde0ef4-service-ca-bundle\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.116674 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.116649 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8d4098-96bf-4828-adba-3a50ea03d86c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.117617 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.117588 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-tmp\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.117717 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.117616 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d18d8dde-86bd-4fff-8a27-a0c0280720a4-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.119192 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.119133 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.119295 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.119275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d18d8dde-86bd-4fff-8a27-a0c0280720a4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.121188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.120705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.121188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.120774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-hub\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.121188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.121025 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-metrics-certs\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.121188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.121154 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7eb76497-1a07-46fa-8b0d-796471c880b9-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m97ws\" (UID: \"7eb76497-1a07-46fa-8b0d-796471c880b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.123072 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.123049 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-ca\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.123072 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.123061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-default-certificate\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.123221 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.123063 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6e8d4098-96bf-4828-adba-3a50ea03d86c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.123392 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.123376 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-96c798986-4m2pt\" (UID: \"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.124068 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.124048 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d50910e2-559c-4954-b35e-d877abde0ef4-stats-auth\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.125978 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.125956 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnbr\" (UniqueName: \"kubernetes.io/projected/6e8d4098-96bf-4828-adba-3a50ea03d86c-kube-api-access-wjnbr\") pod \"cluster-proxy-proxy-agent-d5bf6db6c-6h4dg\" (UID: \"6e8d4098-96bf-4828-adba-3a50ea03d86c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.126468 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.126448 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfwm\" (UniqueName: \"kubernetes.io/projected/d50910e2-559c-4954-b35e-d877abde0ef4-kube-api-access-9pfwm\") pod \"router-default-d7bf988cb-t286m\" (UID: \"d50910e2-559c-4954-b35e-d877abde0ef4\") " pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.126661 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.126643 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mks87\" (UniqueName: \"kubernetes.io/projected/7eb76497-1a07-46fa-8b0d-796471c880b9-kube-api-access-mks87\") pod \"cluster-samples-operator-79b7c55f87-m97ws\" (UID: \"7eb76497-1a07-46fa-8b0d-796471c880b9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.127329 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.127311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5qrt\" (UniqueName: \"kubernetes.io/projected/2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48-kube-api-access-h5qrt\") pod \"managed-serviceaccount-addon-agent-96c798986-4m2pt\" (UID: \"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.131772 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.131607 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:29.132475 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.132453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtp2\" (UniqueName: \"kubernetes.io/projected/d18d8dde-86bd-4fff-8a27-a0c0280720a4-kube-api-access-zbtp2\") pod \"cluster-monitoring-operator-76cf8966f8-fxl5n\" (UID: \"d18d8dde-86bd-4fff-8a27-a0c0280720a4\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.133321 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.133305 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5hh\" (UniqueName: \"kubernetes.io/projected/63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb-kube-api-access-vz5hh\") pod \"klusterlet-addon-workmgr-f68cb858b-5mf4n\" (UID: \"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.201927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.201859 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" Jul 21 14:18:29.216377 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.216489 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216410 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-service-ca-bundle\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.216489 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216439 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fklzd\" (UniqueName: \"kubernetes.io/projected/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-kube-api-access-fklzd\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.216489 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-tmp\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.216649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216519 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-serving-cert\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.216649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216562 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-snapshots\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.216649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216603 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgw5l\" (UniqueName: \"kubernetes.io/projected/5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd-kube-api-access-xgw5l\") pod \"volume-data-source-validator-78cf4c6f6-pq8nz\" (UID: \"5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" Jul 21 14:18:29.216911 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.216885 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-tmp\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.217196 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.217175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-service-ca-bundle\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.217280 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.217260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.217350 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.217326 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-snapshots\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.219448 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.219406 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-serving-cert\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.222095 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.222073 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" Jul 21 14:18:29.227382 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.227364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgw5l\" (UniqueName: \"kubernetes.io/projected/5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd-kube-api-access-xgw5l\") pod \"volume-data-source-validator-78cf4c6f6-pq8nz\" (UID: \"5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" Jul 21 14:18:29.232065 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.232047 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:29.234031 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.233966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fklzd\" (UniqueName: \"kubernetes.io/projected/e6bdcbdf-1506-490a-bd18-64ec3f4ce72a-kube-api-access-fklzd\") pod \"insights-operator-555d9d7885-kglk5\" (UID: \"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a\") " pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.234153 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.234076 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:29.250458 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.250411 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" Jul 21 14:18:29.257105 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.257085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:29.264242 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.264220 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" Jul 21 14:18:29.330308 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.330277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:29.352211 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.352187 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" Jul 21 14:18:29.358975 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.358959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kglk5" Jul 21 14:18:29.430377 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.430283 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:29.505197 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.504376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p447f" event={"ID":"1f9a0b12-10a7-4628-92ed-9d9651940451","Type":"ContainerStarted","Data":"3e6a43fca829a93e5da8140ba8a43da23d0fda28e378cc376bdc2f9536cbf7fa"} Jul 21 14:18:29.533904 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.532280 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p447f" podStartSLOduration=67.07357018 podStartE2EDuration="1m11.532258934s" totalImagesPullingTime="4.458688754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:29.530757526 +0000 UTC m=+71.898267555" watchObservedRunningTime="2026-07-21 14:18:29.532258934 +0000 UTC m=+71.899768965" Jul 21 14:18:29.632136 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.632027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:29.642911 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.642501 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg"] Jul 21 14:18:29.647362 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:29.647113 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e8d4098_96bf_4828_adba_3a50ea03d86c.slice/crio-fbd7869418199ac049963b3b1ca764eff67c80c5d76d0d046b1f47ceac18ef85 WatchSource:0}: Error finding container fbd7869418199ac049963b3b1ca764eff67c80c5d76d0d046b1f47ceac18ef85: Status 404 returned error can't find the container with id fbd7869418199ac049963b3b1ca764eff67c80c5d76d0d046b1f47ceac18ef85 Jul 21 14:18:29.671046 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.669657 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kglk5"] Jul 21 14:18:29.673625 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.673608 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n"] Jul 21 14:18:29.907230 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.907200 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n"] Jul 21 14:18:29.908213 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.908181 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt"] Jul 21 14:18:29.909454 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.909433 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz"] Jul 21 14:18:29.911537 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.911111 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws"] Jul 21 14:18:29.913310 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:29.913286 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d7bf988cb-t286m"] Jul 21 14:18:29.917141 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:29.917076 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50910e2_559c_4954_b35e_d877abde0ef4.slice/crio-e212d7f4d3e1517cfb556bf92f8f257b71c9ea7f3c5c8e00b869c0ef436a2ff3 WatchSource:0}: Error finding container e212d7f4d3e1517cfb556bf92f8f257b71c9ea7f3c5c8e00b869c0ef436a2ff3: Status 404 returned error can't find the container with id e212d7f4d3e1517cfb556bf92f8f257b71c9ea7f3c5c8e00b869c0ef436a2ff3 Jul 21 14:18:30.512973 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.512934 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" event={"ID":"bf728902-c923-4e7a-aeb1-94338a8b1d95","Type":"ContainerStarted","Data":"e5a040c7ec3032025ed9b905ca2638b6d3617f9bc6fb6ee8c5d498c8e47acf70"} Jul 21 14:18:30.514119 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.514097 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" event={"ID":"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48","Type":"ContainerStarted","Data":"b83d22443b7a56b4dd19c7d8f8a0b64db23a6c51128e07816e32b6b862120445"} Jul 21 14:18:30.515248 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.515226 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" event={"ID":"d18d8dde-86bd-4fff-8a27-a0c0280720a4","Type":"ContainerStarted","Data":"f5ae76ed1fe90ca48217b4e9d0dee261b9980e3bdd5f2e6166274d937655b4a9"} Jul 21 14:18:30.516270 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.516253 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kglk5" event={"ID":"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a","Type":"ContainerStarted","Data":"385b4c6b8cad3fcefac6475c4431d81266f51b753778fde6b6ce5ead7fae22ba"} Jul 21 14:18:30.517308 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.517287 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" event={"ID":"7eb76497-1a07-46fa-8b0d-796471c880b9","Type":"ContainerStarted","Data":"ccdc8aacb2b2745e55d6332e0750a61a4044c932d5edd7c22f84aa1a80ac803b"} Jul 21 14:18:30.518283 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.518264 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" event={"ID":"6e8d4098-96bf-4828-adba-3a50ea03d86c","Type":"ContainerStarted","Data":"fbd7869418199ac049963b3b1ca764eff67c80c5d76d0d046b1f47ceac18ef85"} Jul 21 14:18:30.519322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.519305 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" event={"ID":"5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd","Type":"ContainerStarted","Data":"daaa9d0b9186ae7928d8450e042c3b7efa329478864149306433a12ae254bb62"} Jul 21 14:18:30.520301 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.520278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" event={"ID":"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb","Type":"ContainerStarted","Data":"43ebebf41b23f23b037c7814645864e6bc683772bd1f819c34e132f0c3fb5e90"} Jul 21 14:18:30.521835 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.521814 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d7bf988cb-t286m" event={"ID":"d50910e2-559c-4954-b35e-d877abde0ef4","Type":"ContainerStarted","Data":"8cba3d477996832e8e2be791cf0edc8fb96b7c84390435e3e58715e5b28a0538"} Jul 21 14:18:30.521932 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.521845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d7bf988cb-t286m" event={"ID":"d50910e2-559c-4954-b35e-d877abde0ef4","Type":"ContainerStarted","Data":"e212d7f4d3e1517cfb556bf92f8f257b71c9ea7f3c5c8e00b869c0ef436a2ff3"} Jul 21 14:18:30.534120 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.534074 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-jvz68" podStartSLOduration=36.509653006 podStartE2EDuration="41.534056815s" totalImagesPullingTime="5.024403809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:30.532323757 +0000 UTC m=+72.899833787" watchObservedRunningTime="2026-07-21 14:18:30.534056815 +0000 UTC m=+72.901566847" Jul 21 14:18:30.555149 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.554973 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d7bf988cb-t286m" podStartSLOduration=2.554957543 podStartE2EDuration="2.554957543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:30.554029436 +0000 UTC m=+72.921539466" watchObservedRunningTime="2026-07-21 14:18:30.554957543 +0000 UTC m=+72.922467572" Jul 21 14:18:30.703209 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:30.703116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:31.232694 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:31.232613 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:31.235866 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:31.235671 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:31.541298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:31.540705 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:31.542040 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:31.542019 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d7bf988cb-t286m" Jul 21 14:18:33.701411 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:33.701363 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:33.773294 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:33.773260 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:34.005289 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:34.005209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:34.070922 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:34.070856 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:34.135927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:34.135896 2570 scope.go:117] "RemoveContainer" containerID="6727909c7af91ca12d4752aa9a7b9dad74ed32f0defd222c943a25b12659e764" Jul 21 14:18:34.881720 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:34.881685 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:34.956195 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:34.956135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:35.057190 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:35.057107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:36.114416 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:36.114372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:36.175254 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:36.175163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:36.489551 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:36.489522 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-76pfl" Jul 21 14:18:36.865697 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:36.865605 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:36.946000 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:36.945968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:39.086515 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.086437 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:39.182024 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.181957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:39.273527 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.273447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:39.539415 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.539379 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:39.583101 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.583068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" event={"ID":"63a5425a-b7ea-4e01-b8a4-2ac2a0ac74cb","Type":"ContainerStarted","Data":"98f0045e7b71af14bf8d1766302cec6a47f9c0fbb031a39fb23af6bde7920eaa"} Jul 21 14:18:39.583372 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.583346 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:39.584867 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.584846 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" Jul 21 14:18:39.585417 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.585401 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:18:39.585522 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.585503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" event={"ID":"f7f3c68c-9553-4b61-a33f-c30d4e11f9c1","Type":"ContainerStarted","Data":"b85161bd332e60e919269755c0a0a8f574fbe020ba94d189d74ba668f4c1eb83"} Jul 21 14:18:39.585770 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.585754 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:39.587131 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.587109 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" event={"ID":"2d9e6cbc-61b6-4ad9-8f29-8ecc22f68a48","Type":"ContainerStarted","Data":"9a18334c8c2e4a9741cd7673b3650932d7782a8e41fc7743c12e7d46c98b3b1d"} Jul 21 14:18:39.589132 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.588762 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" event={"ID":"d18d8dde-86bd-4fff-8a27-a0c0280720a4","Type":"ContainerStarted","Data":"5e6c443c13952e8f61afea2c3e14eb92fd6a3c55c4574132b36c155e18db3d4d"} Jul 21 14:18:39.591168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.591149 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-kg4q9" Jul 21 14:18:39.591594 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.591375 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kglk5" event={"ID":"e6bdcbdf-1506-490a-bd18-64ec3f4ce72a","Type":"ContainerStarted","Data":"366947d57a8e3ec01ab84a6a17f619672a0824dbf432db24a495e96c4ee9439a"} Jul 21 14:18:39.593373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.593354 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" event={"ID":"7eb76497-1a07-46fa-8b0d-796471c880b9","Type":"ContainerStarted","Data":"36e73c3d16dc199a3742e01044d9d120286f55cd3a613528bfea42cbd258b311"} Jul 21 14:18:39.593475 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.593395 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" event={"ID":"7eb76497-1a07-46fa-8b0d-796471c880b9","Type":"ContainerStarted","Data":"27e39308a3998dd130978a22a4e59387540aca99d29af423df7c4d021d309947"} Jul 21 14:18:39.595959 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.595928 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" event={"ID":"6e8d4098-96bf-4828-adba-3a50ea03d86c","Type":"ContainerStarted","Data":"7533243b400a5606dc8d8aa83b6b9924f988f3bf38db230605e2ab69d1b397ed"} Jul 21 14:18:39.596066 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.595964 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" event={"ID":"6e8d4098-96bf-4828-adba-3a50ea03d86c","Type":"ContainerStarted","Data":"27fc4760e22ca5556218cb30cf73be431e1b6422f7104b308f9f98af67172816"} Jul 21 14:18:39.596066 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.595981 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" event={"ID":"6e8d4098-96bf-4828-adba-3a50ea03d86c","Type":"ContainerStarted","Data":"a6f9107d50d865c057b96ae980e5e5e96ceff2d720835f079c3d6628ddb55099"} Jul 21 14:18:39.597719 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.597698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" event={"ID":"5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd","Type":"ContainerStarted","Data":"057c4878ba0aa5b1bfbf741fbc80abba66465f77e2d30deae3234c0f8923bb3e"} Jul 21 14:18:39.641444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.641394 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-f68cb858b-5mf4n" podStartSLOduration=2.9850911829999998 podStartE2EDuration="11.64137923s" totalImagesPullingTime="8.656288047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.634239276 +0000 UTC m=+82.001749323" watchObservedRunningTime="2026-07-21 14:18:39.64137923 +0000 UTC m=+82.008889236" Jul 21 14:18:39.685990 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.685925 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-pq8nz" podStartSLOduration=2.027663204 podStartE2EDuration="10.685906478s" totalImagesPullingTime="8.658243274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.67795225 +0000 UTC m=+82.045462279" watchObservedRunningTime="2026-07-21 14:18:39.685906478 +0000 UTC m=+82.053416507" Jul 21 14:18:39.714655 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.714623 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v7457"] Jul 21 14:18:39.717395 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.717368 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.725220 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.725200 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 14:18:39.725547 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.725499 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 14:18:39.725727 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.725709 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m7grf\"" Jul 21 14:18:39.745670 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.745647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v7457"] Jul 21 14:18:39.804223 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.804136 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-fxl5n" podStartSLOduration=2.8751560290000002 podStartE2EDuration="11.804118355s" totalImagesPullingTime="8.928962326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.802190714 +0000 UTC m=+82.169700743" watchObservedRunningTime="2026-07-21 14:18:39.804118355 +0000 UTC m=+82.171628382" Jul 21 14:18:39.818715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.818688 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e67ee5-39c5-4e35-94dc-b5db10c90996-crio-socket\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.818860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.818733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2k98\" (UniqueName: \"kubernetes.io/projected/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-api-access-v2k98\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.818860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.818753 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.818860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.818822 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e67ee5-39c5-4e35-94dc-b5db10c90996-data-volume\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.818966 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.818871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.862860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.862811 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-96c798986-4m2pt" podStartSLOduration=3.206027778 podStartE2EDuration="11.862796083s" totalImagesPullingTime="8.656768305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.861856335 +0000 UTC m=+82.229366388" watchObservedRunningTime="2026-07-21 14:18:39.862796083 +0000 UTC m=+82.230306107" Jul 21 14:18:39.881517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.881467 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-kglk5" podStartSLOduration=1.986844306 podStartE2EDuration="10.881448718s" totalImagesPullingTime="8.894604412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.88035053 +0000 UTC m=+82.247860557" watchObservedRunningTime="2026-07-21 14:18:39.881448718 +0000 UTC m=+82.248958746" Jul 21 14:18:39.899131 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.899082 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m97ws" podStartSLOduration=3.329126717 podStartE2EDuration="11.899064229s" totalImagesPullingTime="8.569937512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.898613399 +0000 UTC m=+82.266123427" watchObservedRunningTime="2026-07-21 14:18:39.899064229 +0000 UTC m=+82.266574251" Jul 21 14:18:39.920002 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.919972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e67ee5-39c5-4e35-94dc-b5db10c90996-crio-socket\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.920164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.920033 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2k98\" (UniqueName: \"kubernetes.io/projected/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-api-access-v2k98\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.920164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.920074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.920164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.920107 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a3e67ee5-39c5-4e35-94dc-b5db10c90996-crio-socket\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.920164 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:39.920152 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 14:18:39.920346 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:39.920215 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm podName:a3e67ee5-39c5-4e35-94dc-b5db10c90996 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:40.4201971 +0000 UTC m=+82.787707108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v7457" (UID: "a3e67ee5-39c5-4e35-94dc-b5db10c90996") : configmap "kube-rbac-proxy" not found Jul 21 14:18:39.920346 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.920247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e67ee5-39c5-4e35-94dc-b5db10c90996-data-volume\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.920346 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.920325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.920457 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:39.920437 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 14:18:39.920491 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:39.920476 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls podName:a3e67ee5-39c5-4e35-94dc-b5db10c90996 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:40.42046638 +0000 UTC m=+82.787976386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v7457" (UID: "a3e67ee5-39c5-4e35-94dc-b5db10c90996") : secret "insights-runtime-extractor-tls" not found Jul 21 14:18:39.920569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.920552 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a3e67ee5-39c5-4e35-94dc-b5db10c90996-data-volume\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:39.924713 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.924670 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d5bf6db6c-6h4dg" podStartSLOduration=3.001246368 podStartE2EDuration="11.924657164s" totalImagesPullingTime="8.923410796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:39.924181336 +0000 UTC m=+82.291691363" watchObservedRunningTime="2026-07-21 14:18:39.924657164 +0000 UTC m=+82.292167214" Jul 21 14:18:39.931906 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:39.931883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2k98\" (UniqueName: \"kubernetes.io/projected/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-api-access-v2k98\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:40.229536 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:40.229509 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76pfl_e444c9d9-48ba-4131-9b16-2c8e26296df0/dns/0.log" Jul 21 14:18:40.424810 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:40.424772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:40.424953 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:40.424865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:40.424953 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:40.424929 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 14:18:40.425072 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:40.424982 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 14:18:40.425072 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:40.424999 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls podName:a3e67ee5-39c5-4e35-94dc-b5db10c90996 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:41.424984464 +0000 UTC m=+83.792494470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v7457" (UID: "a3e67ee5-39c5-4e35-94dc-b5db10c90996") : secret "insights-runtime-extractor-tls" not found Jul 21 14:18:40.425072 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:40.425058 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm podName:a3e67ee5-39c5-4e35-94dc-b5db10c90996 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:41.425044508 +0000 UTC m=+83.792554515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v7457" (UID: "a3e67ee5-39c5-4e35-94dc-b5db10c90996") : configmap "kube-rbac-proxy" not found Jul 21 14:18:40.430399 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:40.430377 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76pfl_e444c9d9-48ba-4131-9b16-2c8e26296df0/kube-rbac-proxy/0.log" Jul 21 14:18:41.434987 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:41.434942 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:41.435403 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:41.435047 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:41.435403 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:41.435082 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 14:18:41.435403 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:41.435156 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm podName:a3e67ee5-39c5-4e35-94dc-b5db10c90996 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:43.435140362 +0000 UTC m=+85.802650368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm") pod "insights-runtime-extractor-v7457" (UID: "a3e67ee5-39c5-4e35-94dc-b5db10c90996") : configmap "kube-rbac-proxy" not found Jul 21 14:18:41.435403 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:41.435206 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 14:18:41.435403 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:41.435260 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls podName:a3e67ee5-39c5-4e35-94dc-b5db10c90996 nodeName:}" failed. No retries permitted until 2026-07-21 14:18:43.435247196 +0000 UTC m=+85.802757202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls") pod "insights-runtime-extractor-v7457" (UID: "a3e67ee5-39c5-4e35-94dc-b5db10c90996") : secret "insights-runtime-extractor-tls" not found Jul 21 14:18:41.629535 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:41.629507 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g442s_292c96a4-7f66-4fca-adfc-8a86176f342a/dns-node-resolver/0.log" Jul 21 14:18:42.231439 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:42.231410 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-846d7479f5-jpdkd_808270b1-884c-43ed-ab2e-4f9ab3102332/registry/0.log" Jul 21 14:18:42.431089 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:42.431062 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fcf8cfd48-sjz45_d7e477da-8762-4d12-b6ad-7cfceec3dd1f/registry/0.log" Jul 21 14:18:42.628729 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:42.628644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:42.757700 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:42.757667 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:42.831073 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:42.831044 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7t9g8_f91b710d-ac22-48bd-b037-439391d77258/node-ca/0.log" Jul 21 14:18:43.229765 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.229737 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7bf988cb-t286m_d50910e2-559c-4954-b35e-d877abde0ef4/router/0.log" Jul 21 14:18:43.453227 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.453196 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:43.453391 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.453248 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:43.453784 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.453752 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a3e67ee5-39c5-4e35-94dc-b5db10c90996-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:43.455489 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.455466 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a3e67ee5-39c5-4e35-94dc-b5db10c90996-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7457\" (UID: \"a3e67ee5-39c5-4e35-94dc-b5db10c90996\") " pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:43.628046 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.627951 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v7457" Jul 21 14:18:43.634437 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.634415 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xp7xl_d9f27890-994f-491e-b14e-52319993fc85/serve-healthcheck-canary/0.log" Jul 21 14:18:43.763199 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:43.763179 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v7457"] Jul 21 14:18:43.765151 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:18:43.765126 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e67ee5_39c5_4e35_94dc_b5db10c90996.slice/crio-2d88d2dbe49d3c663e9f2909d6c78b20a43bf90ba8760858cab5f5aec6c22e55 WatchSource:0}: Error finding container 2d88d2dbe49d3c663e9f2909d6c78b20a43bf90ba8760858cab5f5aec6c22e55: Status 404 returned error can't find the container with id 2d88d2dbe49d3c663e9f2909d6c78b20a43bf90ba8760858cab5f5aec6c22e55 Jul 21 14:18:44.191022 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:44.190967 2570 patch_prober.go:28] interesting pod/image-registry-846d7479f5-jpdkd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 14:18:44.191215 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:44.191057 2570 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" podUID="808270b1-884c-43ed-ab2e-4f9ab3102332" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 14:18:44.618108 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:44.618023 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7457" event={"ID":"a3e67ee5-39c5-4e35-94dc-b5db10c90996","Type":"ContainerStarted","Data":"2d88d2dbe49d3c663e9f2909d6c78b20a43bf90ba8760858cab5f5aec6c22e55"} Jul 21 14:18:45.479686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:45.479643 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:18:45.667029 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:45.666982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:45.767936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:45.767846 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:45.864724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:45.864667 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:47.331452 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:47.331416 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:47.498392 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:47.498342 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:47.597808 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:47.597625 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:48.636545 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:48.636508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7457" event={"ID":"a3e67ee5-39c5-4e35-94dc-b5db10c90996","Type":"ContainerStarted","Data":"2c279e7da088ca00c45a984525275da5e1ebcb3912435316b1e9e95e0689890e"} Jul 21 14:18:48.976042 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:48.975992 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:51.214135 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:51.214100 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:51.315462 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:51.315415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:51.829444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:51.829404 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:51.931413 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:51.931388 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:52.396509 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:52.396451 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:52.466389 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:52.466346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:52.656708 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:52.656617 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7457" event={"ID":"a3e67ee5-39c5-4e35-94dc-b5db10c90996","Type":"ContainerStarted","Data":"d65bfa872c0cb9bce94bc75e5e3252e08dd3be10968987917bbd7d4ad9b80241"} Jul 21 14:18:53.989835 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:53.989786 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" podUID="d7e477da-8762-4d12-b6ad-7cfceec3dd1f" containerName="registry" containerID="cri-o://8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480" gracePeriod=30 Jul 21 14:18:54.639978 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:54.639925 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:54.757645 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:54.757616 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:55.387163 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.387137 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:55.448050 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448018 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-installation-pull-secrets\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448215 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448067 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s4ls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-kube-api-access-2s4ls\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448215 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448124 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-image-registry-private-configuration\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448215 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448171 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-bound-sa-token\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448215 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448206 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448416 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448242 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-certificates\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448416 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448273 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-trusted-ca\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448416 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448300 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-ca-trust-extracted\") pod \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\" (UID: \"d7e477da-8762-4d12-b6ad-7cfceec3dd1f\") " Jul 21 14:18:55.448830 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448791 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:18:55.448987 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.448948 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:18:55.450867 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.450818 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:18:55.450867 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.450847 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:18:55.451038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.450907 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:18:55.451038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.450984 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:18:55.451038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.450988 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-kube-api-access-2s4ls" (OuterVolumeSpecName: "kube-api-access-2s4ls") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "kube-api-access-2s4ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:18:55.456211 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.456189 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d7e477da-8762-4d12-b6ad-7cfceec3dd1f" (UID: "d7e477da-8762-4d12-b6ad-7cfceec3dd1f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:18:55.549361 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549285 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-installation-pull-secrets\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549361 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549313 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s4ls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-kube-api-access-2s4ls\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549361 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549325 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-image-registry-private-configuration\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549361 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549334 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-bound-sa-token\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549361 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549343 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-tls\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549361 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549362 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-registry-certificates\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549637 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549371 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-trusted-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.549637 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.549379 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7e477da-8762-4d12-b6ad-7cfceec3dd1f-ca-trust-extracted\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:18:55.670467 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.670427 2570 generic.go:356] "Generic (PLEG): container finished" podID="d7e477da-8762-4d12-b6ad-7cfceec3dd1f" containerID="8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480" exitCode=0 Jul 21 14:18:55.670640 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.670471 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" Jul 21 14:18:55.670640 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.670514 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" event={"ID":"d7e477da-8762-4d12-b6ad-7cfceec3dd1f","Type":"ContainerDied","Data":"8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480"} Jul 21 14:18:55.670640 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.670562 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fcf8cfd48-sjz45" event={"ID":"d7e477da-8762-4d12-b6ad-7cfceec3dd1f","Type":"ContainerDied","Data":"1bd40a64625ec926d8ea92721992a308299827cff01323db7bccd1da116953db"} Jul 21 14:18:55.670640 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.670586 2570 scope.go:117] "RemoveContainer" containerID="8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480" Jul 21 14:18:55.673054 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.673028 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7457" event={"ID":"a3e67ee5-39c5-4e35-94dc-b5db10c90996","Type":"ContainerStarted","Data":"681db23271b1f7db4dff84741e1d5321c07d7bdf8906257c7f0dfcbca4bd781e"} Jul 21 14:18:55.680767 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.680752 2570 scope.go:117] "RemoveContainer" containerID="8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480" Jul 21 14:18:55.681056 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:18:55.681034 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480\": container with ID starting with 8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480 not found: ID does not exist" containerID="8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480" Jul 21 14:18:55.681118 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.681064 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480"} err="failed to get container status \"8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480\": rpc error: code = NotFound desc = could not find container \"8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480\": container with ID starting with 8a3c9bb7003aaee2ed5edd3bd85c95653c9e8a116a8edbe3333323de4d511480 not found: ID does not exist" Jul 21 14:18:55.696369 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.696326 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v7457" podStartSLOduration=5.275679511 podStartE2EDuration="16.696312449s" totalImagesPullingTime="11.420632938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 14:18:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:18:55.694786446 +0000 UTC m=+98.062296474" watchObservedRunningTime="2026-07-21 14:18:55.696312449 +0000 UTC m=+98.063822476" Jul 21 14:18:55.717551 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.717527 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fcf8cfd48-sjz45"] Jul 21 14:18:55.719329 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.719307 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fcf8cfd48-sjz45"] Jul 21 14:18:55.761066 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.760999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:55.851958 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.851874 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:55.949048 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:55.948999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:56.021282 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:56.021245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:56.140627 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:56.140594 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7e477da-8762-4d12-b6ad-7cfceec3dd1f" path="/var/lib/kubelet/pods/d7e477da-8762-4d12-b6ad-7cfceec3dd1f/volumes" Jul 21 14:18:56.697121 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:56.697088 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846d7479f5-jpdkd"] Jul 21 14:18:56.787171 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:56.787143 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:56.888405 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:56.888376 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:56.988068 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:56.987976 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:57.489180 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:57.489156 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mr7zj" Jul 21 14:18:58.906934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:58.906892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:58.981956 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:58.981925 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:59.189884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:59.189801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:18:59.263083 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:18:59.263050 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:03.635455 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:03.635423 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:03.702394 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:03.702360 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:04.186159 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:04.186122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:05.796397 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:05.796357 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:05.892286 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:05.892252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:06.557186 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:06.557156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:06.998553 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:06.998523 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r4dmd"] Jul 21 14:19:06.998957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:06.998907 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7e477da-8762-4d12-b6ad-7cfceec3dd1f" containerName="registry" Jul 21 14:19:06.998957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:06.998921 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7e477da-8762-4d12-b6ad-7cfceec3dd1f" containerName="registry" Jul 21 14:19:06.999074 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:06.999021 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7e477da-8762-4d12-b6ad-7cfceec3dd1f" containerName="registry" Jul 21 14:19:07.005240 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.005218 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.012265 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.012243 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 14:19:07.012571 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.012265 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 14:19:07.012571 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.012273 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 14:19:07.013478 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.013344 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 14:19:07.013478 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.013380 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nlq6z\"" Jul 21 14:19:07.028074 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028055 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-root\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028082 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-tls\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028108 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-wtmp\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028293 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf3f586-f2c5-43a5-a749-96e81395e317-metrics-client-ca\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028293 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028231 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-textfile\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028293 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028403 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028319 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028403 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028346 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-sys\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.028403 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.028369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwrx\" (UniqueName: \"kubernetes.io/projected/5bf3f586-f2c5-43a5-a749-96e81395e317-kube-api-access-zlwrx\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.054961 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.054936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:07.129451 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129429 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-sys\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwrx\" (UniqueName: \"kubernetes.io/projected/5bf3f586-f2c5-43a5-a749-96e81395e317-kube-api-access-zlwrx\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-root\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129527 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-sys\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129726 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:19:07.129620 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 14:19:07.129726 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-tls\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129726 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:19:07.129683 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-tls podName:5bf3f586-f2c5-43a5-a749-96e81395e317 nodeName:}" failed. No retries permitted until 2026-07-21 14:19:07.629664527 +0000 UTC m=+109.997174537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-tls") pod "node-exporter-r4dmd" (UID: "5bf3f586-f2c5-43a5-a749-96e81395e317") : secret "node-exporter-tls" not found Jul 21 14:19:07.129726 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129721 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-wtmp\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129720 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-root\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf3f586-f2c5-43a5-a749-96e81395e317-metrics-client-ca\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-textfile\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-wtmp\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129833 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.129935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.129889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.130225 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.130110 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-textfile\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.130325 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.130306 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bf3f586-f2c5-43a5-a749-96e81395e317-metrics-client-ca\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.130452 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.130433 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-accelerators-collector-config\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.132485 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.132466 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.146218 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.146194 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwrx\" (UniqueName: \"kubernetes.io/projected/5bf3f586-f2c5-43a5-a749-96e81395e317-kube-api-access-zlwrx\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.153307 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.153281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:07.252232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.251897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:07.634969 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.634902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-tls\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.637084 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.637062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bf3f586-f2c5-43a5-a749-96e81395e317-node-exporter-tls\") pod \"node-exporter-r4dmd\" (UID: \"5bf3f586-f2c5-43a5-a749-96e81395e317\") " pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.914932 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:07.914908 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r4dmd" Jul 21 14:19:07.974129 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:19:07.974104 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf3f586_f2c5_43a5_a749_96e81395e317.slice/crio-83e5bf4a4674304cb0621fa4eecf78bffa47ab47e1ce259af7a43fc70063a60c WatchSource:0}: Error finding container 83e5bf4a4674304cb0621fa4eecf78bffa47ab47e1ce259af7a43fc70063a60c: Status 404 returned error can't find the container with id 83e5bf4a4674304cb0621fa4eecf78bffa47ab47e1ce259af7a43fc70063a60c Jul 21 14:19:08.724388 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:08.724344 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4dmd" event={"ID":"5bf3f586-f2c5-43a5-a749-96e81395e317","Type":"ContainerStarted","Data":"83e5bf4a4674304cb0621fa4eecf78bffa47ab47e1ce259af7a43fc70063a60c"} Jul 21 14:19:09.609474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:09.609410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:09.678067 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:09.678039 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:09.729603 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:09.729576 2570 generic.go:356] "Generic (PLEG): container finished" podID="5bf3f586-f2c5-43a5-a749-96e81395e317" containerID="2bf2c23628a01f7f382d545011dc8e7d8f6a17dab4fd7fc39109efbff73dcb72" exitCode=0 Jul 21 14:19:09.729895 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:09.729648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4dmd" event={"ID":"5bf3f586-f2c5-43a5-a749-96e81395e317","Type":"ContainerDied","Data":"2bf2c23628a01f7f382d545011dc8e7d8f6a17dab4fd7fc39109efbff73dcb72"} Jul 21 14:19:09.963421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:09.963389 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:10.037652 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:10.037618 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:10.134052 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:10.133974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:10.200989 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:10.200967 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:10.735793 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:10.735764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4dmd" event={"ID":"5bf3f586-f2c5-43a5-a749-96e81395e317","Type":"ContainerStarted","Data":"d7fc7773efc4bc7c993efb8b59e2f6df884477cde686051566c0cc507cfda38f"} Jul 21 14:19:10.736230 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:10.735820 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r4dmd" event={"ID":"5bf3f586-f2c5-43a5-a749-96e81395e317","Type":"ContainerStarted","Data":"c482fe04f8c318edacb201170f83619ae68f1154235a8f45e4f5e5dccb3a08c9"} Jul 21 14:19:10.761080 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:10.761040 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r4dmd" podStartSLOduration=3.469034602 podStartE2EDuration="4.761025414s" totalImagesPullingTime="1.207771669s" totalInitContainerRuntime="84.219143ms" isStatefulPod=true podCreationTimestamp="2026-07-21 14:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:19:10.759082874 +0000 UTC m=+113.126592902" watchObservedRunningTime="2026-07-21 14:19:10.761025414 +0000 UTC m=+113.128535443" Jul 21 14:19:11.558802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:11.558747 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:11.653716 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:11.653681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:11.879643 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:11.879565 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:12.590526 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:12.590497 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:12.690856 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:12.690821 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:13.268943 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:13.268907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:14.793041 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:14.792991 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:21.716261 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:21.716211 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" podUID="808270b1-884c-43ed-ab2e-4f9ab3102332" containerName="registry" containerID="cri-o://f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d" gracePeriod=30 Jul 21 14:19:21.778373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:21.778339 2570 generic.go:356] "Generic (PLEG): container finished" podID="795f58c3-8685-44d3-a1f0-bf1a3bed6ca6" containerID="75e1f3cd23c4d9dbc3d285fb5cdeb7fbc162981009ef1cc791a6a7c022a669c2" exitCode=0 Jul 21 14:19:21.778502 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:21.778410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" event={"ID":"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6","Type":"ContainerDied","Data":"75e1f3cd23c4d9dbc3d285fb5cdeb7fbc162981009ef1cc791a6a7c022a669c2"} Jul 21 14:19:21.778746 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:21.778731 2570 scope.go:117] "RemoveContainer" containerID="75e1f3cd23c4d9dbc3d285fb5cdeb7fbc162981009ef1cc791a6a7c022a669c2" Jul 21 14:19:21.826104 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:21.826069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:22.015224 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.015196 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:19:22.026687 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.026659 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:22.135207 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135134 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dlvb\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-kube-api-access-9dlvb\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135207 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135173 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-image-registry-private-configuration\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135449 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135237 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135449 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135275 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/808270b1-884c-43ed-ab2e-4f9ab3102332-ca-trust-extracted\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135449 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135302 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-trusted-ca\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135449 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135326 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-certificates\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135449 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135353 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-installation-pull-secrets\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.135726 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.135502 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-bound-sa-token\") pod \"808270b1-884c-43ed-ab2e-4f9ab3102332\" (UID: \"808270b1-884c-43ed-ab2e-4f9ab3102332\") " Jul 21 14:19:22.136781 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.136750 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:19:22.137240 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.137215 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:19:22.137798 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.137776 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:19:22.138151 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.138129 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-kube-api-access-9dlvb" (OuterVolumeSpecName: "kube-api-access-9dlvb") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "kube-api-access-9dlvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:19:22.138289 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.138267 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:19:22.138437 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.138412 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:19:22.140342 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.140313 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:19:22.146276 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.146254 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/808270b1-884c-43ed-ab2e-4f9ab3102332-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "808270b1-884c-43ed-ab2e-4f9ab3102332" (UID: "808270b1-884c-43ed-ab2e-4f9ab3102332"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:19:22.236566 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236527 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-tls\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236566 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236559 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/808270b1-884c-43ed-ab2e-4f9ab3102332-ca-trust-extracted\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236566 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236573 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-trusted-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236584 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/808270b1-884c-43ed-ab2e-4f9ab3102332-registry-certificates\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236596 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-installation-pull-secrets\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236609 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-bound-sa-token\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236619 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dlvb\" (UniqueName: \"kubernetes.io/projected/808270b1-884c-43ed-ab2e-4f9ab3102332-kube-api-access-9dlvb\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.236786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.236632 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/808270b1-884c-43ed-ab2e-4f9ab3102332-image-registry-private-configuration\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:19:22.289724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.289683 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:22.389529 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.389493 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:22.449907 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.449871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:22.645615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.645531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:22.788052 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.787996 2570 generic.go:356] "Generic (PLEG): container finished" podID="808270b1-884c-43ed-ab2e-4f9ab3102332" containerID="f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d" exitCode=0 Jul 21 14:19:22.788483 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.788071 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" event={"ID":"808270b1-884c-43ed-ab2e-4f9ab3102332","Type":"ContainerDied","Data":"f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d"} Jul 21 14:19:22.788483 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.788147 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" event={"ID":"808270b1-884c-43ed-ab2e-4f9ab3102332","Type":"ContainerDied","Data":"4dc0e6c852282bcc0c9bdd66cecf082acaefdb8afbdd4c1ded4a332c5d0a9588"} Jul 21 14:19:22.788483 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.788079 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846d7479f5-jpdkd" Jul 21 14:19:22.788483 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.788169 2570 scope.go:117] "RemoveContainer" containerID="f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d" Jul 21 14:19:22.790184 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.790157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-5d5xd" event={"ID":"795f58c3-8685-44d3-a1f0-bf1a3bed6ca6","Type":"ContainerStarted","Data":"5eb83e8196a2306cf7a475e3e44903fe4b365c99fedb509103420ea818e56f81"} Jul 21 14:19:22.799951 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.799582 2570 scope.go:117] "RemoveContainer" containerID="f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d" Jul 21 14:19:22.800206 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:19:22.800175 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d\": container with ID starting with f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d not found: ID does not exist" containerID="f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d" Jul 21 14:19:22.800303 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.800213 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d"} err="failed to get container status \"f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d\": rpc error: code = NotFound desc = could not find container \"f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d\": container with ID starting with f86fb6731deb96fd1471a3f63811a1e7389b55b794ba0ba1ee173aea3b228a7d not found: ID does not exist" Jul 21 14:19:22.833970 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.833940 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846d7479f5-jpdkd"] Jul 21 14:19:22.843524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:22.843497 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-846d7479f5-jpdkd"] Jul 21 14:19:24.139842 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:24.139807 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808270b1-884c-43ed-ab2e-4f9ab3102332" path="/var/lib/kubelet/pods/808270b1-884c-43ed-ab2e-4f9ab3102332/volumes" Jul 21 14:19:28.993707 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:28.993671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:29.057742 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:29.057705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:29.442727 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:29.442687 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:29.543563 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:29.543530 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:29.644851 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:29.644816 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:32.828544 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:32.828506 2570 generic.go:356] "Generic (PLEG): container finished" podID="5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2" containerID="51051325d964c1aec7e498a7d34b7fb0b1d2a7f4a18922cdf30b6e1571c78910" exitCode=0 Jul 21 14:19:32.828936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:32.828583 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" event={"ID":"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2","Type":"ContainerDied","Data":"51051325d964c1aec7e498a7d34b7fb0b1d2a7f4a18922cdf30b6e1571c78910"} Jul 21 14:19:32.828936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:32.828900 2570 scope.go:117] "RemoveContainer" containerID="51051325d964c1aec7e498a7d34b7fb0b1d2a7f4a18922cdf30b6e1571c78910" Jul 21 14:19:33.834182 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:33.834151 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kdxvv" event={"ID":"5a0aa87c-bc2b-4b5c-93e8-11d991b31fd2","Type":"ContainerStarted","Data":"cec43a9a32e4772b08ebc418c441a22f4831d2a93f08da751ef487cc1bd81a04"} Jul 21 14:19:42.451038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:42.450990 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:19:42.536373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:19:42.536334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:09.981671 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:09.981629 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:22.992073 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:22.992027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:23.093445 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:23.093396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:25.635755 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:25.635720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:32.062274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:32.062229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:32.147826 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:32.147797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:34.996613 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:34.996578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:35.096902 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:35.096869 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:43.350108 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:43.350058 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:43.935606 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:43.935559 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:44.000842 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:44.000789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:51.823287 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:51.823240 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:51.928001 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:51.927961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:20:51.993778 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:20:51.993739 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:21:17.129838 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:21:17.129768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:21:17.258919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:21:17.258873 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:22:18.044605 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:22:18.044571 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:22:18.050356 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:22:18.050316 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:22:18.056151 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:22:18.056124 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 14:22:44.700273 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:22:44.700229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:22:47.077781 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:22:47.077742 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:02.742421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:02.742375 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:03.566140 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:03.566103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:06.550987 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:06.550949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:06.651654 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:06.651624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:07.655845 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:07.655806 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:07.755853 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:07.755823 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:08.534421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:08.534382 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:08.629883 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:08.629852 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:10.596936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:10.596842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:10.948896 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:10.948857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:13.379559 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:13.379514 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:13.478786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:13.478749 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:18.350154 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:18.350117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:18.450538 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:18.450510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:19.795849 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:19.795812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:24.935244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:24.935209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:25.044904 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:25.044872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:27.029066 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:27.029028 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:39.135297 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:39.135253 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:39.231390 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:39.231355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:44.749556 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:44.749519 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:44.848879 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:44.848847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:45.307254 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.307223 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:45.408772 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.408715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:45.714615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.714548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:45.829289 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.829259 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:45.856841 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.856811 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn"] Jul 21 14:23:45.857129 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.857116 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808270b1-884c-43ed-ab2e-4f9ab3102332" containerName="registry" Jul 21 14:23:45.857176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.857130 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="808270b1-884c-43ed-ab2e-4f9ab3102332" containerName="registry" Jul 21 14:23:45.857210 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.857188 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="808270b1-884c-43ed-ab2e-4f9ab3102332" containerName="registry" Jul 21 14:23:45.860129 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.860114 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:45.865213 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.865195 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 14:23:45.865324 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.865221 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 14:23:45.865451 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.865435 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n8qqh\"" Jul 21 14:23:45.866310 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.866296 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 14:23:45.866558 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.866542 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 14:23:45.882989 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:45.882966 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn"] Jul 21 14:23:46.009626 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.009547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1224731d-6898-403b-9a72-061884710a78-manifests\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.009626 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.009581 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1224731d-6898-403b-9a72-061884710a78-webhook-cert\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.009626 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.009618 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1224731d-6898-403b-9a72-061884710a78-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.009860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.009689 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1224731d-6898-403b-9a72-061884710a78-tmp\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.009860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.009747 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxsn\" (UniqueName: \"kubernetes.io/projected/1224731d-6898-403b-9a72-061884710a78-kube-api-access-gqxsn\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.110048 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.109999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1224731d-6898-403b-9a72-061884710a78-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.110188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.110059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1224731d-6898-403b-9a72-061884710a78-tmp\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.110188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.110094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxsn\" (UniqueName: \"kubernetes.io/projected/1224731d-6898-403b-9a72-061884710a78-kube-api-access-gqxsn\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.110188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.110128 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1224731d-6898-403b-9a72-061884710a78-manifests\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.110188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.110156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1224731d-6898-403b-9a72-061884710a78-webhook-cert\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.110562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.110535 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1224731d-6898-403b-9a72-061884710a78-manifests\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.111073 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.111031 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:46.112538 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.112515 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1224731d-6898-403b-9a72-061884710a78-tmp\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.112993 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.112966 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1224731d-6898-403b-9a72-061884710a78-webhook-cert\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.113120 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.113101 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1224731d-6898-403b-9a72-061884710a78-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.121636 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.121614 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxsn\" (UniqueName: \"kubernetes.io/projected/1224731d-6898-403b-9a72-061884710a78-kube-api-access-gqxsn\") pod \"opendatahub-operator-controller-manager-86bb54d755-6wlhn\" (UID: \"1224731d-6898-403b-9a72-061884710a78\") " pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.170049 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.170029 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:46.210643 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.210604 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:46.300551 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.300478 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn"] Jul 21 14:23:46.304174 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:23:46.304150 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1224731d_6898_403b_9a72_061884710a78.slice/crio-1c0821e98b9fd2a2053c82130101ae17dcccf164cf87ffd2ca76e0b2623a4121 WatchSource:0}: Error finding container 1c0821e98b9fd2a2053c82130101ae17dcccf164cf87ffd2ca76e0b2623a4121: Status 404 returned error can't find the container with id 1c0821e98b9fd2a2053c82130101ae17dcccf164cf87ffd2ca76e0b2623a4121 Jul 21 14:23:46.306159 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.306126 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:46.306265 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.306210 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:23:46.392196 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.392157 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:46.545162 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.545134 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:46.646114 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.646086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:23:46.768791 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:46.768759 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" event={"ID":"1224731d-6898-403b-9a72-061884710a78","Type":"ContainerStarted","Data":"1c0821e98b9fd2a2053c82130101ae17dcccf164cf87ffd2ca76e0b2623a4121"} Jul 21 14:23:49.785313 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:49.785249 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" event={"ID":"1224731d-6898-403b-9a72-061884710a78","Type":"ContainerStarted","Data":"ddb6cbec90e45058de48c9be745639111fed0d91a5b08908ccff94b5225ce322"} Jul 21 14:23:50.790821 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:50.790789 2570 generic.go:356] "Generic (PLEG): container finished" podID="1224731d-6898-403b-9a72-061884710a78" containerID="ddb6cbec90e45058de48c9be745639111fed0d91a5b08908ccff94b5225ce322" exitCode=0 Jul 21 14:23:50.791157 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:50.790837 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" event={"ID":"1224731d-6898-403b-9a72-061884710a78","Type":"ContainerDied","Data":"ddb6cbec90e45058de48c9be745639111fed0d91a5b08908ccff94b5225ce322"} Jul 21 14:23:51.797390 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:51.797355 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" event={"ID":"1224731d-6898-403b-9a72-061884710a78","Type":"ContainerStarted","Data":"25c9c03c4a8b2fc35d7c6d77c54f63f1ae12ffd0c6368ecaa386685288d48be9"} Jul 21 14:23:51.797772 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:51.797545 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:23:51.824476 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:23:51.824425 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" podStartSLOduration=2.06811707 podStartE2EDuration="6.824406668s" totalImagesPullingTime="3.704917112s" totalInitContainerRuntime="1.051372486s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:23:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:23:51.82271665 +0000 UTC m=+394.190226678" watchObservedRunningTime="2026-07-21 14:23:51.824406668 +0000 UTC m=+394.191916696" Jul 21 14:24:02.803970 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:02.803935 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86bb54d755-6wlhn" Jul 21 14:24:06.095015 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.094965 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:06.207612 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.207568 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:06.509680 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.509637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:06.636586 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.636552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:06.735534 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.735499 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5bbd655967-vc576"] Jul 21 14:24:06.744831 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.744800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.747907 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.747881 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 14:24:06.748042 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.747911 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 14:24:06.752040 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.751998 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bbd655967-vc576"] Jul 21 14:24:06.755042 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.754999 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cfb9n\"" Jul 21 14:24:06.870112 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.870037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1079e15-b066-49c9-9476-a14d12ff7182-tmp\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.870278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.870175 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzzn\" (UniqueName: \"kubernetes.io/projected/d1079e15-b066-49c9-9476-a14d12ff7182-kube-api-access-7kzzn\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.870278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.870211 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1079e15-b066-49c9-9476-a14d12ff7182-tls-certs\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.895762 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.895726 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z4zgl"] Jul 21 14:24:06.898094 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.898068 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:06.901177 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.901154 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 14:24:06.901407 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.901171 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2kfvd\"" Jul 21 14:24:06.918695 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.918662 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z4zgl"] Jul 21 14:24:06.959813 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.959767 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:06.970955 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.970916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1079e15-b066-49c9-9476-a14d12ff7182-tmp\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.971148 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.971037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzzn\" (UniqueName: \"kubernetes.io/projected/d1079e15-b066-49c9-9476-a14d12ff7182-kube-api-access-7kzzn\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.971148 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.971071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1079e15-b066-49c9-9476-a14d12ff7182-tls-certs\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.973750 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.973725 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1079e15-b066-49c9-9476-a14d12ff7182-tmp\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.973948 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.973931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1079e15-b066-49c9-9476-a14d12ff7182-tls-certs\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:06.981550 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:06.981520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzzn\" (UniqueName: \"kubernetes.io/projected/d1079e15-b066-49c9-9476-a14d12ff7182-kube-api-access-7kzzn\") pod \"kube-auth-proxy-5bbd655967-vc576\" (UID: \"d1079e15-b066-49c9-9476-a14d12ff7182\") " pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:07.056232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.056193 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" Jul 21 14:24:07.072029 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.071970 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqd5z\" (UniqueName: \"kubernetes.io/projected/216dd99e-221f-4ab3-b954-d7fc4f43954c-kube-api-access-vqd5z\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.072163 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.072068 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216dd99e-221f-4ab3-b954-d7fc4f43954c-cert\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.074309 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.074278 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:07.174489 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.174455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqd5z\" (UniqueName: \"kubernetes.io/projected/216dd99e-221f-4ab3-b954-d7fc4f43954c-kube-api-access-vqd5z\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.174919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.174505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216dd99e-221f-4ab3-b954-d7fc4f43954c-cert\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.174919 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:24:07.174678 2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 14:24:07.174919 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:24:07.174751 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/216dd99e-221f-4ab3-b954-d7fc4f43954c-cert podName:216dd99e-221f-4ab3-b954-d7fc4f43954c nodeName:}" failed. No retries permitted until 2026-07-21 14:24:07.674728394 +0000 UTC m=+410.042238402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/216dd99e-221f-4ab3-b954-d7fc4f43954c-cert") pod "odh-model-controller-7998c67dc4-z4zgl" (UID: "216dd99e-221f-4ab3-b954-d7fc4f43954c") : secret "odh-model-controller-webhook-cert" not found Jul 21 14:24:07.185125 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.185097 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqd5z\" (UniqueName: \"kubernetes.io/projected/216dd99e-221f-4ab3-b954-d7fc4f43954c-kube-api-access-vqd5z\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.212585 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.212557 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5bbd655967-vc576"] Jul 21 14:24:07.215112 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:24:07.215083 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1079e15_b066_49c9_9476_a14d12ff7182.slice/crio-78fadc651b727af5796e4cc30d3617c33755683bc9f729ece9666c7a49bfb77a WatchSource:0}: Error finding container 78fadc651b727af5796e4cc30d3617c33755683bc9f729ece9666c7a49bfb77a: Status 404 returned error can't find the container with id 78fadc651b727af5796e4cc30d3617c33755683bc9f729ece9666c7a49bfb77a Jul 21 14:24:07.323245 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.323213 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:07.497383 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.497285 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:07.515334 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.515305 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dswlc"] Jul 21 14:24:07.518633 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.518613 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.522217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.522193 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 14:24:07.532186 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.532163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dswlc"] Jul 21 14:24:07.679176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.679133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216dd99e-221f-4ab3-b954-d7fc4f43954c-cert\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.679373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.679199 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfff\" (UniqueName: \"kubernetes.io/projected/874af69e-95c2-4b04-8124-fa68f9a0e8ce-kube-api-access-wcfff\") pod \"model-serving-api-54d89bd5fc-dswlc\" (UID: \"874af69e-95c2-4b04-8124-fa68f9a0e8ce\") " pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.679373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.679236 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/874af69e-95c2-4b04-8124-fa68f9a0e8ce-tls-certs\") pod \"model-serving-api-54d89bd5fc-dswlc\" (UID: \"874af69e-95c2-4b04-8124-fa68f9a0e8ce\") " pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.681695 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.681671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/216dd99e-221f-4ab3-b954-d7fc4f43954c-cert\") pod \"odh-model-controller-7998c67dc4-z4zgl\" (UID: \"216dd99e-221f-4ab3-b954-d7fc4f43954c\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.706157 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.706118 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:07.779836 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.779740 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfff\" (UniqueName: \"kubernetes.io/projected/874af69e-95c2-4b04-8124-fa68f9a0e8ce-kube-api-access-wcfff\") pod \"model-serving-api-54d89bd5fc-dswlc\" (UID: \"874af69e-95c2-4b04-8124-fa68f9a0e8ce\") " pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.779836 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.779773 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/874af69e-95c2-4b04-8124-fa68f9a0e8ce-tls-certs\") pod \"model-serving-api-54d89bd5fc-dswlc\" (UID: \"874af69e-95c2-4b04-8124-fa68f9a0e8ce\") " pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.782322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.782301 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/874af69e-95c2-4b04-8124-fa68f9a0e8ce-tls-certs\") pod \"model-serving-api-54d89bd5fc-dswlc\" (UID: \"874af69e-95c2-4b04-8124-fa68f9a0e8ce\") " pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.789879 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.789857 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfff\" (UniqueName: \"kubernetes.io/projected/874af69e-95c2-4b04-8124-fa68f9a0e8ce-kube-api-access-wcfff\") pod \"model-serving-api-54d89bd5fc-dswlc\" (UID: \"874af69e-95c2-4b04-8124-fa68f9a0e8ce\") " pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.809488 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.809468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:07.829128 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.828910 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:07.874027 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.873434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" event={"ID":"d1079e15-b066-49c9-9476-a14d12ff7182","Type":"ContainerStarted","Data":"78fadc651b727af5796e4cc30d3617c33755683bc9f729ece9666c7a49bfb77a"} Jul 21 14:24:07.988580 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:07.987139 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z4zgl"] Jul 21 14:24:07.992129 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:24:07.991510 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216dd99e_221f_4ab3_b954_d7fc4f43954c.slice/crio-e4d382ee2fad2b409c9b9127a9cb9376086153fc680209e837826b5a34c2a1d1 WatchSource:0}: Error finding container e4d382ee2fad2b409c9b9127a9cb9376086153fc680209e837826b5a34c2a1d1: Status 404 returned error can't find the container with id e4d382ee2fad2b409c9b9127a9cb9376086153fc680209e837826b5a34c2a1d1 Jul 21 14:24:08.030561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.030541 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dswlc"] Jul 21 14:24:08.032658 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:24:08.032635 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874af69e_95c2_4b04_8124_fa68f9a0e8ce.slice/crio-e49d3f2d9ef09ff814084f7a2df77fa91d857de96d393b11d08b944d2506abe1 WatchSource:0}: Error finding container e49d3f2d9ef09ff814084f7a2df77fa91d857de96d393b11d08b944d2506abe1: Status 404 returned error can't find the container with id e49d3f2d9ef09ff814084f7a2df77fa91d857de96d393b11d08b944d2506abe1 Jul 21 14:24:08.367867 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.367753 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:08.375941 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.375915 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc"] Jul 21 14:24:08.378816 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.378793 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.381517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.381493 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 14:24:08.381517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.381510 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 14:24:08.382445 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.382033 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 14:24:08.382445 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.382308 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 14:24:08.382622 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.382467 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dxlr4\"" Jul 21 14:24:08.395630 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.395608 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc"] Jul 21 14:24:08.466725 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.466695 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:08.484841 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.484806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.485035 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.484875 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-token\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.485035 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.484941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.485035 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.484999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7dw\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-kube-api-access-7l7dw\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.485217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.485079 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-local-certs\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.485217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.485108 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.485217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.485160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-cacerts\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586555 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586521 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-token\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7dw\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-kube-api-access-7l7dw\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586693 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-local-certs\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-cacerts\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.586988 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.586753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.587383 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.587336 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.589341 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.589319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-cacerts\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.589455 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.589328 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-local-certs\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.589455 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.589362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.589611 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.589600 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.597793 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.597767 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-token\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.598869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.598846 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7dw\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-kube-api-access-7l7dw\") pod \"istiod-openshift-gateway-7574459656-ddzdc\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.691824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.691788 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:08.865859 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.865830 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc"] Jul 21 14:24:08.869539 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:24:08.869505 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cc83d1_ae14_4aac_8e94_4d37a39bf2ce.slice/crio-0527ea428e7ce033ccf052a7523168261961de961b67d3723c2cfd71160832d3 WatchSource:0}: Error finding container 0527ea428e7ce033ccf052a7523168261961de961b67d3723c2cfd71160832d3: Status 404 returned error can't find the container with id 0527ea428e7ce033ccf052a7523168261961de961b67d3723c2cfd71160832d3 Jul 21 14:24:08.885730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.885695 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" event={"ID":"874af69e-95c2-4b04-8124-fa68f9a0e8ce","Type":"ContainerStarted","Data":"e49d3f2d9ef09ff814084f7a2df77fa91d857de96d393b11d08b944d2506abe1"} Jul 21 14:24:08.888999 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.888973 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" event={"ID":"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce","Type":"ContainerStarted","Data":"0527ea428e7ce033ccf052a7523168261961de961b67d3723c2cfd71160832d3"} Jul 21 14:24:08.891665 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:08.891606 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" event={"ID":"216dd99e-221f-4ab3-b954-d7fc4f43954c","Type":"ContainerStarted","Data":"e4d382ee2fad2b409c9b9127a9cb9376086153fc680209e837826b5a34c2a1d1"} Jul 21 14:24:09.623506 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:09.623459 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:13.041441 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.041397 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 14:24:13.041798 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.041506 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 14:24:13.919075 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.918961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:13.920440 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.920415 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" event={"ID":"874af69e-95c2-4b04-8124-fa68f9a0e8ce","Type":"ContainerStarted","Data":"a164cafff464b08e628da353cbd97af4aa650f8f326023aed75e3162afff8bc8"} Jul 21 14:24:13.920582 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.920508 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:13.922228 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.922204 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" event={"ID":"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce","Type":"ContainerStarted","Data":"0d043374500d399c41cde53ab65347352926c75d5f24afff25189668903fb478"} Jul 21 14:24:13.922366 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.922252 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:13.923746 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.923727 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" event={"ID":"d1079e15-b066-49c9-9476-a14d12ff7182","Type":"ContainerStarted","Data":"c921535c4491b627b9588335c9b364537e76c0e2ff10a62bc07428e28e361596"} Jul 21 14:24:13.925297 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.925266 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" event={"ID":"216dd99e-221f-4ab3-b954-d7fc4f43954c","Type":"ContainerStarted","Data":"38c3737b0eda4f55572b9e13be361b369c3d0e1c887b950d028a24574e3de5f0"} Jul 21 14:24:13.925383 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.925331 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:13.946807 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.946761 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" podStartSLOduration=2.003743586 podStartE2EDuration="6.94674706s" totalImagesPullingTime="4.943003474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:24:13.944660955 +0000 UTC m=+416.312170988" watchObservedRunningTime="2026-07-21 14:24:13.94674706 +0000 UTC m=+416.314257087" Jul 21 14:24:13.968190 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.968113 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5bbd655967-vc576" podStartSLOduration=2.211470041 podStartE2EDuration="7.968090477s" totalImagesPullingTime="5.756620436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:24:13.966413948 +0000 UTC m=+416.333923997" watchObservedRunningTime="2026-07-21 14:24:13.968090477 +0000 UTC m=+416.335600506" Jul 21 14:24:13.991090 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:13.991031 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" podStartSLOduration=1.821891236 podStartE2EDuration="5.990992888s" totalImagesPullingTime="4.169101652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:24:13.990092337 +0000 UTC m=+416.357602365" watchObservedRunningTime="2026-07-21 14:24:13.990992888 +0000 UTC m=+416.358502917" Jul 21 14:24:14.000384 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:14.000345 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:14.008310 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:14.008261 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" podStartSLOduration=3.013564573 podStartE2EDuration="8.00824913s" totalImagesPullingTime="4.994684557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:24:14.006940489 +0000 UTC m=+416.374450528" watchObservedRunningTime="2026-07-21 14:24:14.00824913 +0000 UTC m=+416.375759157" Jul 21 14:24:14.715058 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:14.714994 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:14.931686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:14.931661 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:14.960641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:14.960608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:15.145397 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:15.145355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:18.536156 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.536117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:18.600743 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.600712 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-trj7w"] Jul 21 14:24:18.605620 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.605598 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.608675 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.608649 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 14:24:18.608785 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.608651 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2fgkd\"" Jul 21 14:24:18.615566 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.615546 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-trj7w"] Jul 21 14:24:18.632870 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.632845 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:18.671598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.671562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c250ff-e40d-49fd-94cf-d66c1f7c7960-cert\") pod \"kserve-controller-manager-d6f4b65b8-trj7w\" (UID: \"43c250ff-e40d-49fd-94cf-d66c1f7c7960\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.671740 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.671610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb4c4\" (UniqueName: \"kubernetes.io/projected/43c250ff-e40d-49fd-94cf-d66c1f7c7960-kube-api-access-sb4c4\") pod \"kserve-controller-manager-d6f4b65b8-trj7w\" (UID: \"43c250ff-e40d-49fd-94cf-d66c1f7c7960\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.772838 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.772813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c250ff-e40d-49fd-94cf-d66c1f7c7960-cert\") pod \"kserve-controller-manager-d6f4b65b8-trj7w\" (UID: \"43c250ff-e40d-49fd-94cf-d66c1f7c7960\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.772955 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.772848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb4c4\" (UniqueName: \"kubernetes.io/projected/43c250ff-e40d-49fd-94cf-d66c1f7c7960-kube-api-access-sb4c4\") pod \"kserve-controller-manager-d6f4b65b8-trj7w\" (UID: \"43c250ff-e40d-49fd-94cf-d66c1f7c7960\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.775262 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.775238 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c250ff-e40d-49fd-94cf-d66c1f7c7960-cert\") pod \"kserve-controller-manager-d6f4b65b8-trj7w\" (UID: \"43c250ff-e40d-49fd-94cf-d66c1f7c7960\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.783893 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.783873 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb4c4\" (UniqueName: \"kubernetes.io/projected/43c250ff-e40d-49fd-94cf-d66c1f7c7960-kube-api-access-sb4c4\") pod \"kserve-controller-manager-d6f4b65b8-trj7w\" (UID: \"43c250ff-e40d-49fd-94cf-d66c1f7c7960\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:18.918535 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:18.918509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:19.050581 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:19.050549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:19.061987 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:19.061952 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-trj7w"] Jul 21 14:24:19.065714 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:24:19.065673 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c250ff_e40d_49fd_94cf_d66c1f7c7960.slice/crio-5d71546f315d51d2afa4c12416c2fa34f0ac5ba2cd758f78f6b88b9eb5ef73a1 WatchSource:0}: Error finding container 5d71546f315d51d2afa4c12416c2fa34f0ac5ba2cd758f78f6b88b9eb5ef73a1: Status 404 returned error can't find the container with id 5d71546f315d51d2afa4c12416c2fa34f0ac5ba2cd758f78f6b88b9eb5ef73a1 Jul 21 14:24:19.185172 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:19.185105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:19.951912 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:19.951879 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" event={"ID":"43c250ff-e40d-49fd-94cf-d66c1f7c7960","Type":"ContainerStarted","Data":"5d71546f315d51d2afa4c12416c2fa34f0ac5ba2cd758f78f6b88b9eb5ef73a1"} Jul 21 14:24:21.963263 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:21.963189 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" event={"ID":"43c250ff-e40d-49fd-94cf-d66c1f7c7960","Type":"ContainerStarted","Data":"3a4ad30650aca62aaebfee7047e1d90f0b9b8d9c9f5fb2e3c9efe4db9b48d9f8"} Jul 21 14:24:21.963612 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:21.963312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:24:21.985384 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:21.985345 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" podStartSLOduration=1.3832123649999999 podStartE2EDuration="3.985333316s" totalImagesPullingTime="2.602120951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:24:21.984700988 +0000 UTC m=+424.352211015" watchObservedRunningTime="2026-07-21 14:24:21.985333316 +0000 UTC m=+424.352843344" Jul 21 14:24:22.767586 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:22.767528 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc"] Jul 21 14:24:22.767745 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:22.767723 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" podUID="48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" containerName="discovery" containerID="cri-o://0d043374500d399c41cde53ab65347352926c75d5f24afff25189668903fb478" gracePeriod=30 Jul 21 14:24:22.969887 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:22.969859 2570 generic.go:356] "Generic (PLEG): container finished" podID="48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" containerID="0d043374500d399c41cde53ab65347352926c75d5f24afff25189668903fb478" exitCode=0 Jul 21 14:24:22.970221 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:22.969933 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" event={"ID":"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce","Type":"ContainerDied","Data":"0d043374500d399c41cde53ab65347352926c75d5f24afff25189668903fb478"} Jul 21 14:24:23.006642 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.006620 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:23.115122 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-token\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115122 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115106 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-ca-configmap\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115138 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-local-certs\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115176 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-dns-cert\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115207 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-kubeconfig\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115253 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l7dw\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-kube-api-access-7l7dw\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115286 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-cacerts\") pod \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\" (UID: \"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce\") " Jul 21 14:24:23.115556 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115511 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:24:23.115888 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.115857 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-ca-configmap\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.117496 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.117470 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-cacerts" (OuterVolumeSpecName: "cacerts") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:24:23.117496 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.117484 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:24:23.117717 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.117690 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-kube-api-access-7l7dw" (OuterVolumeSpecName: "kube-api-access-7l7dw") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "kube-api-access-7l7dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:24:23.117774 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.117756 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:24:23.117870 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.117849 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:24:23.117909 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.117849 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-local-certs" (OuterVolumeSpecName: "local-certs") pod "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" (UID: "48cc83d1-ae14-4aac-8e94-4d37a39bf2ce"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:24:23.216784 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.216748 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7dw\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-kube-api-access-7l7dw\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.216784 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.216782 2570 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-cacerts\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.216784 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.216791 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-token\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.216937 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.216800 2570 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-local-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.216937 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.216809 2570 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-csr-dns-cert\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.216937 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.216817 2570 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce-istio-kubeconfig\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:24:23.975888 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.975851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" event={"ID":"48cc83d1-ae14-4aac-8e94-4d37a39bf2ce","Type":"ContainerDied","Data":"0527ea428e7ce033ccf052a7523168261961de961b67d3723c2cfd71160832d3"} Jul 21 14:24:23.975888 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.975880 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc" Jul 21 14:24:23.976417 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:23.975909 2570 scope.go:117] "RemoveContainer" containerID="0d043374500d399c41cde53ab65347352926c75d5f24afff25189668903fb478" Jul 21 14:24:24.002365 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:24.002341 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc"] Jul 21 14:24:24.006967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:24.006939 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-ddzdc"] Jul 21 14:24:24.140220 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:24.140191 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" path="/var/lib/kubelet/pods/48cc83d1-ae14-4aac-8e94-4d37a39bf2ce/volumes" Jul 21 14:24:24.932176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:24.932148 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-z4zgl" Jul 21 14:24:24.935200 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:24.935181 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dswlc" Jul 21 14:24:32.143660 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:32.143621 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:32.210569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:32.210537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:32.658323 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:32.658288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:32.739055 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:32.739020 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:43.197045 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:43.196955 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:45.970998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:45.970958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:46.042892 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:46.042849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:24:52.974709 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:24:52.974679 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-trj7w" Jul 21 14:25:07.710815 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.710779 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d9zcf"] Jul 21 14:25:07.711208 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.711124 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" containerName="discovery" Jul 21 14:25:07.711208 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.711136 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" containerName="discovery" Jul 21 14:25:07.711279 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.711215 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="48cc83d1-ae14-4aac-8e94-4d37a39bf2ce" containerName="discovery" Jul 21 14:25:07.723300 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.723276 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:07.724833 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.724806 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d9zcf"] Jul 21 14:25:07.725920 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.725898 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 14:25:07.726063 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.725943 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dnbhp\"" Jul 21 14:25:07.727242 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.727221 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 14:25:07.768839 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.768814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:07.769343 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.769325 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbltb\" (UniqueName: \"kubernetes.io/projected/8c4d5ddb-f57f-497b-bf84-7500da3fb72d-kube-api-access-pbltb\") pod \"kuadrant-operator-catalog-d9zcf\" (UID: \"8c4d5ddb-f57f-497b-bf84-7500da3fb72d\") " pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:07.870106 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.870085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbltb\" (UniqueName: \"kubernetes.io/projected/8c4d5ddb-f57f-497b-bf84-7500da3fb72d-kube-api-access-pbltb\") pod \"kuadrant-operator-catalog-d9zcf\" (UID: \"8c4d5ddb-f57f-497b-bf84-7500da3fb72d\") " pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:07.882390 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:07.882366 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbltb\" (UniqueName: \"kubernetes.io/projected/8c4d5ddb-f57f-497b-bf84-7500da3fb72d-kube-api-access-pbltb\") pod \"kuadrant-operator-catalog-d9zcf\" (UID: \"8c4d5ddb-f57f-497b-bf84-7500da3fb72d\") " pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:08.034026 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.033924 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:08.055060 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.055021 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d9zcf"] Jul 21 14:25:08.116720 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.116673 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:08.164445 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.164416 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d9zcf"] Jul 21 14:25:08.167479 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:25:08.167451 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4d5ddb_f57f_497b_bf84_7500da3fb72d.slice/crio-949f807f08293fe663e0d21fb36cf0858172e71c4a5313424a0485762daf24bd WatchSource:0}: Error finding container 949f807f08293fe663e0d21fb36cf0858172e71c4a5313424a0485762daf24bd: Status 404 returned error can't find the container with id 949f807f08293fe663e0d21fb36cf0858172e71c4a5313424a0485762daf24bd Jul 21 14:25:08.176900 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.176875 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" event={"ID":"8c4d5ddb-f57f-497b-bf84-7500da3fb72d","Type":"ContainerStarted","Data":"949f807f08293fe663e0d21fb36cf0858172e71c4a5313424a0485762daf24bd"} Jul 21 14:25:08.266736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.266712 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bpsr"] Jul 21 14:25:08.271148 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.271134 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:08.273397 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.273379 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdpw\" (UniqueName: \"kubernetes.io/projected/c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a-kube-api-access-lsdpw\") pod \"kuadrant-operator-catalog-6bpsr\" (UID: \"c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a\") " pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:08.280224 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.280204 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bpsr"] Jul 21 14:25:08.374266 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.374211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdpw\" (UniqueName: \"kubernetes.io/projected/c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a-kube-api-access-lsdpw\") pod \"kuadrant-operator-catalog-6bpsr\" (UID: \"c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a\") " pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:08.384599 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.384577 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdpw\" (UniqueName: \"kubernetes.io/projected/c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a-kube-api-access-lsdpw\") pod \"kuadrant-operator-catalog-6bpsr\" (UID: \"c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a\") " pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:08.580651 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.580623 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:08.710724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:08.710690 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bpsr"] Jul 21 14:25:08.713213 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:25:08.713187 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f31b9e_5ef3_4926_a318_b9cc7cf1d45a.slice/crio-f43fee01cb3c92457b9649a1f8a7e3a28e0b82071bb0a359995dfac6cd9cf196 WatchSource:0}: Error finding container f43fee01cb3c92457b9649a1f8a7e3a28e0b82071bb0a359995dfac6cd9cf196: Status 404 returned error can't find the container with id f43fee01cb3c92457b9649a1f8a7e3a28e0b82071bb0a359995dfac6cd9cf196 Jul 21 14:25:09.184516 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:09.184480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" event={"ID":"c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a","Type":"ContainerStarted","Data":"f43fee01cb3c92457b9649a1f8a7e3a28e0b82071bb0a359995dfac6cd9cf196"} Jul 21 14:25:11.195879 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.195803 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" event={"ID":"c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a","Type":"ContainerStarted","Data":"6f64f857e13cf5800b63ee37538c2908edfa28073c1cbfe263c308d719333c7e"} Jul 21 14:25:11.197649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.197619 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" event={"ID":"8c4d5ddb-f57f-497b-bf84-7500da3fb72d","Type":"ContainerStarted","Data":"bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140"} Jul 21 14:25:11.197781 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.197659 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" podUID="8c4d5ddb-f57f-497b-bf84-7500da3fb72d" containerName="registry-server" containerID="cri-o://bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140" gracePeriod=2 Jul 21 14:25:11.221215 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.221174 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" podStartSLOduration=0.868179439 podStartE2EDuration="3.221152401s" totalImagesPullingTime="2.352972962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:25:11.21893374 +0000 UTC m=+473.586443767" watchObservedRunningTime="2026-07-21 14:25:11.221152401 +0000 UTC m=+473.588662443" Jul 21 14:25:11.238650 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.238611 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" podStartSLOduration=1.3427101989999999 podStartE2EDuration="4.238597894s" totalImagesPullingTime="2.895887695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:25:11.238046556 +0000 UTC m=+473.605556585" watchObservedRunningTime="2026-07-21 14:25:11.238597894 +0000 UTC m=+473.606107921" Jul 21 14:25:11.434458 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.434438 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d9zcf_8c4d5ddb-f57f-497b-bf84-7500da3fb72d/registry-server/0.log" Jul 21 14:25:11.436772 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.436755 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:11.497886 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.497862 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbltb\" (UniqueName: \"kubernetes.io/projected/8c4d5ddb-f57f-497b-bf84-7500da3fb72d-kube-api-access-pbltb\") pod \"8c4d5ddb-f57f-497b-bf84-7500da3fb72d\" (UID: \"8c4d5ddb-f57f-497b-bf84-7500da3fb72d\") " Jul 21 14:25:11.499819 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.499793 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4d5ddb-f57f-497b-bf84-7500da3fb72d-kube-api-access-pbltb" (OuterVolumeSpecName: "kube-api-access-pbltb") pod "8c4d5ddb-f57f-497b-bf84-7500da3fb72d" (UID: "8c4d5ddb-f57f-497b-bf84-7500da3fb72d"). InnerVolumeSpecName "kube-api-access-pbltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:25:11.599269 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:11.599210 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbltb\" (UniqueName: \"kubernetes.io/projected/8c4d5ddb-f57f-497b-bf84-7500da3fb72d-kube-api-access-pbltb\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:25:12.202468 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.202442 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d9zcf_8c4d5ddb-f57f-497b-bf84-7500da3fb72d/registry-server/0.log" Jul 21 14:25:12.202818 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.202484 2570 generic.go:356] "Generic (PLEG): container finished" podID="8c4d5ddb-f57f-497b-bf84-7500da3fb72d" containerID="bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140" exitCode=1 Jul 21 14:25:12.202818 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.202528 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" Jul 21 14:25:12.202818 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.202569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" event={"ID":"8c4d5ddb-f57f-497b-bf84-7500da3fb72d","Type":"ContainerDied","Data":"bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140"} Jul 21 14:25:12.202818 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.202614 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d9zcf" event={"ID":"8c4d5ddb-f57f-497b-bf84-7500da3fb72d","Type":"ContainerDied","Data":"949f807f08293fe663e0d21fb36cf0858172e71c4a5313424a0485762daf24bd"} Jul 21 14:25:12.202818 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.202635 2570 scope.go:117] "RemoveContainer" containerID="bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140" Jul 21 14:25:12.211409 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.211393 2570 scope.go:117] "RemoveContainer" containerID="bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140" Jul 21 14:25:12.211689 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:25:12.211671 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140\": container with ID starting with bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140 not found: ID does not exist" containerID="bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140" Jul 21 14:25:12.211733 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.211697 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140"} err="failed to get container status \"bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140\": rpc error: code = NotFound desc = could not find container \"bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140\": container with ID starting with bd1ce5081e53c26372fdd5fd1059764d5bf6b62b302fc08f240b51cd2dd35140 not found: ID does not exist" Jul 21 14:25:12.223656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.223635 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d9zcf"] Jul 21 14:25:12.230422 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:12.230404 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d9zcf"] Jul 21 14:25:14.141200 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:14.141158 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4d5ddb-f57f-497b-bf84-7500da3fb72d" path="/var/lib/kubelet/pods/8c4d5ddb-f57f-497b-bf84-7500da3fb72d/volumes" Jul 21 14:25:18.581442 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:18.581395 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:18.581916 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:18.581468 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:18.603247 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:18.603226 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:19.257782 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:19.257755 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6bpsr" Jul 21 14:25:26.688173 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:26.688130 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:35.048066 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:35.048029 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:35.151476 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:35.151441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:37.847662 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:37.847601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:39.856771 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.856737 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k"] Jul 21 14:25:39.857169 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.857154 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4d5ddb-f57f-497b-bf84-7500da3fb72d" containerName="registry-server" Jul 21 14:25:39.857169 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.857171 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4d5ddb-f57f-497b-bf84-7500da3fb72d" containerName="registry-server" Jul 21 14:25:39.857269 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.857259 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4d5ddb-f57f-497b-bf84-7500da3fb72d" containerName="registry-server" Jul 21 14:25:39.861574 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.861557 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:39.864308 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.864287 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-7k2dw\"" Jul 21 14:25:39.871936 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.871915 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k"] Jul 21 14:25:39.918967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:39.918936 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52bc\" (UniqueName: \"kubernetes.io/projected/41ed6ab1-8979-4bf2-be37-1e0f750dfe59-kube-api-access-g52bc\") pod \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" (UID: \"41ed6ab1-8979-4bf2-be37-1e0f750dfe59\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:40.019637 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.019600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52bc\" (UniqueName: \"kubernetes.io/projected/41ed6ab1-8979-4bf2-be37-1e0f750dfe59-kube-api-access-g52bc\") pod \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" (UID: \"41ed6ab1-8979-4bf2-be37-1e0f750dfe59\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:40.030994 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.030970 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52bc\" (UniqueName: \"kubernetes.io/projected/41ed6ab1-8979-4bf2-be37-1e0f750dfe59-kube-api-access-g52bc\") pod \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" (UID: \"41ed6ab1-8979-4bf2-be37-1e0f750dfe59\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:40.171730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.171701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:40.257083 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.257049 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:40.309783 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.309752 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k"] Jul 21 14:25:40.313075 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:25:40.313044 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ed6ab1_8979_4bf2_be37_1e0f750dfe59.slice/crio-136f7c0d06b3ce186348e4acae1cd610469d3d52ca1a4a0e94eaa4e092147cb1 WatchSource:0}: Error finding container 136f7c0d06b3ce186348e4acae1cd610469d3d52ca1a4a0e94eaa4e092147cb1: Status 404 returned error can't find the container with id 136f7c0d06b3ce186348e4acae1cd610469d3d52ca1a4a0e94eaa4e092147cb1 Jul 21 14:25:40.329067 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.329038 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" event={"ID":"41ed6ab1-8979-4bf2-be37-1e0f750dfe59","Type":"ContainerStarted","Data":"136f7c0d06b3ce186348e4acae1cd610469d3d52ca1a4a0e94eaa4e092147cb1"} Jul 21 14:25:40.606927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:40.606847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:41.432295 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:41.432262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:42.340325 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:42.340291 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" event={"ID":"41ed6ab1-8979-4bf2-be37-1e0f750dfe59","Type":"ContainerStarted","Data":"c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1"} Jul 21 14:25:42.340478 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:42.340399 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:42.362577 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:42.362527 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" podStartSLOduration=1.414294157 podStartE2EDuration="3.362512444s" totalImagesPullingTime="1.948218287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:25:42.361163039 +0000 UTC m=+504.728673067" watchObservedRunningTime="2026-07-21 14:25:42.362512444 +0000 UTC m=+504.730022471" Jul 21 14:25:44.143440 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.143407 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d"] Jul 21 14:25:44.146758 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.146736 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:44.149267 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.149246 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tmxsq\"" Jul 21 14:25:44.158452 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.158429 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d"] Jul 21 14:25:44.256696 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.256673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chd9g\" (UniqueName: \"kubernetes.io/projected/ae07fb97-090c-4368-a75d-20676aa30d08-kube-api-access-chd9g\") pod \"authorino-operator-75cc8d5cb7-hrx5d\" (UID: \"ae07fb97-090c-4368-a75d-20676aa30d08\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:44.357230 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.357207 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chd9g\" (UniqueName: \"kubernetes.io/projected/ae07fb97-090c-4368-a75d-20676aa30d08-kube-api-access-chd9g\") pod \"authorino-operator-75cc8d5cb7-hrx5d\" (UID: \"ae07fb97-090c-4368-a75d-20676aa30d08\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:44.371043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.371021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chd9g\" (UniqueName: \"kubernetes.io/projected/ae07fb97-090c-4368-a75d-20676aa30d08-kube-api-access-chd9g\") pod \"authorino-operator-75cc8d5cb7-hrx5d\" (UID: \"ae07fb97-090c-4368-a75d-20676aa30d08\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:44.456634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.456582 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:44.594519 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.594374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d"] Jul 21 14:25:44.597497 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:25:44.597458 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae07fb97_090c_4368_a75d_20676aa30d08.slice/crio-700758a9c06dc817174d2924809b228ea320027860e6f2e52f01bc086274c29d WatchSource:0}: Error finding container 700758a9c06dc817174d2924809b228ea320027860e6f2e52f01bc086274c29d: Status 404 returned error can't find the container with id 700758a9c06dc817174d2924809b228ea320027860e6f2e52f01bc086274c29d Jul 21 14:25:44.867194 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.867112 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:44.964291 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:44.964255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:45.264562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:45.264523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:45.354436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:45.354408 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" event={"ID":"ae07fb97-090c-4368-a75d-20676aa30d08","Type":"ContainerStarted","Data":"700758a9c06dc817174d2924809b228ea320027860e6f2e52f01bc086274c29d"} Jul 21 14:25:46.664100 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:46.664060 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:47.366618 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.366581 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" event={"ID":"ae07fb97-090c-4368-a75d-20676aa30d08","Type":"ContainerStarted","Data":"a8fda210f166e1adda54bbb7a09110da4114f236f5c130afe532ca0769ef2acc"} Jul 21 14:25:47.366803 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.366667 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:47.390161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.390111 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" podStartSLOduration=0.766972904 podStartE2EDuration="3.390096721s" totalImagesPullingTime="2.623123817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:25:47.388864661 +0000 UTC m=+509.756374688" watchObservedRunningTime="2026-07-21 14:25:47.390096721 +0000 UTC m=+509.757606748" Jul 21 14:25:47.559969 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.559901 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr"] Jul 21 14:25:47.563207 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.563191 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:25:47.565689 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.565667 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 14:25:47.565793 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.565724 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cf982\"" Jul 21 14:25:47.576040 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.576020 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr"] Jul 21 14:25:47.687144 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.687118 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2sq\" (UniqueName: \"kubernetes.io/projected/e31ff0c9-264c-4660-8936-a3d35c3dde6c-kube-api-access-tk2sq\") pod \"dns-operator-controller-manager-864dd78c4c-82pkr\" (UID: \"e31ff0c9-264c-4660-8936-a3d35c3dde6c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:25:47.788323 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.788296 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2sq\" (UniqueName: \"kubernetes.io/projected/e31ff0c9-264c-4660-8936-a3d35c3dde6c-kube-api-access-tk2sq\") pod \"dns-operator-controller-manager-864dd78c4c-82pkr\" (UID: \"e31ff0c9-264c-4660-8936-a3d35c3dde6c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:25:47.800121 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.800094 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2sq\" (UniqueName: \"kubernetes.io/projected/e31ff0c9-264c-4660-8936-a3d35c3dde6c-kube-api-access-tk2sq\") pod \"dns-operator-controller-manager-864dd78c4c-82pkr\" (UID: \"e31ff0c9-264c-4660-8936-a3d35c3dde6c\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:25:47.872801 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:47.872724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:25:48.008663 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:48.008635 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr"] Jul 21 14:25:48.011749 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:25:48.011725 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31ff0c9_264c_4660_8936_a3d35c3dde6c.slice/crio-5009ed711d023070317c9a436ddb217d241874c5fbc39a018ff2aa13eefceb2c WatchSource:0}: Error finding container 5009ed711d023070317c9a436ddb217d241874c5fbc39a018ff2aa13eefceb2c: Status 404 returned error can't find the container with id 5009ed711d023070317c9a436ddb217d241874c5fbc39a018ff2aa13eefceb2c Jul 21 14:25:48.154820 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:48.154786 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:48.372544 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:48.372509 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" event={"ID":"e31ff0c9-264c-4660-8936-a3d35c3dde6c","Type":"ContainerStarted","Data":"5009ed711d023070317c9a436ddb217d241874c5fbc39a018ff2aa13eefceb2c"} Jul 21 14:25:49.068327 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:49.068243 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:50.385588 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:50.385551 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" event={"ID":"e31ff0c9-264c-4660-8936-a3d35c3dde6c","Type":"ContainerStarted","Data":"67a282584132aff8e8ac9aae29270a31a313eb1ca7c98c2ede69113e95835e6a"} Jul 21 14:25:50.386083 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:50.385628 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:25:50.405902 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:50.405856 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" podStartSLOduration=1.227696264 podStartE2EDuration="3.405841627s" totalImagesPullingTime="2.178145363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:25:50.404587582 +0000 UTC m=+512.772097609" watchObservedRunningTime="2026-07-21 14:25:50.405841627 +0000 UTC m=+512.773351654" Jul 21 14:25:52.494048 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.493993 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:52.529574 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.529542 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp"] Jul 21 14:25:52.532904 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.532886 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.535715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.535694 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-82jnm\"" Jul 21 14:25:52.548046 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.547994 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp"] Jul 21 14:25:52.627795 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.627767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0f35e7-65dd-4b26-878f-2f001df66ae7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.627932 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.627810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq62x\" (UniqueName: \"kubernetes.io/projected/ce0f35e7-65dd-4b26-878f-2f001df66ae7-kube-api-access-xq62x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.729096 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.729063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq62x\" (UniqueName: \"kubernetes.io/projected/ce0f35e7-65dd-4b26-878f-2f001df66ae7-kube-api-access-xq62x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.729253 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.729164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0f35e7-65dd-4b26-878f-2f001df66ae7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.729525 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.729508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0f35e7-65dd-4b26-878f-2f001df66ae7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.741528 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.741498 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq62x\" (UniqueName: \"kubernetes.io/projected/ce0f35e7-65dd-4b26-878f-2f001df66ae7-kube-api-access-xq62x\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:52.843114 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:52.843037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:53.004641 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:25:53.004605 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0f35e7_65dd_4b26_878f_2f001df66ae7.slice/crio-44a8023f5285fa84479b4fd1d65910e38c71aea4e52e02fef46312b34a828818 WatchSource:0}: Error finding container 44a8023f5285fa84479b4fd1d65910e38c71aea4e52e02fef46312b34a828818: Status 404 returned error can't find the container with id 44a8023f5285fa84479b4fd1d65910e38c71aea4e52e02fef46312b34a828818 Jul 21 14:25:53.017666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:53.017641 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp"] Jul 21 14:25:53.347647 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:53.347625 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:25:53.401050 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:53.400992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" event={"ID":"ce0f35e7-65dd-4b26-878f-2f001df66ae7","Type":"ContainerStarted","Data":"44a8023f5285fa84479b4fd1d65910e38c71aea4e52e02fef46312b34a828818"} Jul 21 14:25:54.056700 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:54.056664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:54.157317 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:54.157280 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:58.086587 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:58.086544 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:58.189785 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:58.189749 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:58.375747 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:58.375672 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hrx5d" Jul 21 14:25:58.568863 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:58.568831 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:58.689139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:58.689108 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:25:59.432268 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:59.432241 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" event={"ID":"ce0f35e7-65dd-4b26-878f-2f001df66ae7","Type":"ContainerStarted","Data":"843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f"} Jul 21 14:25:59.432662 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:59.432353 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:25:59.488357 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:25:59.488311 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" podStartSLOduration=1.909981859 podStartE2EDuration="7.488298819s" totalImagesPullingTime="5.57831696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:25:59.488164669 +0000 UTC m=+521.855674698" watchObservedRunningTime="2026-07-21 14:25:59.488298819 +0000 UTC m=+521.855808846" Jul 21 14:26:01.393224 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:01.393196 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-82pkr" Jul 21 14:26:01.585943 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:01.585902 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:01.726950 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:01.726878 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:01.837404 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:01.837338 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:06.211883 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:06.211855 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:06.312913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:06.312878 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:10.438643 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:10.438610 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.200706 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.200671 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp"] Jul 21 14:26:12.201179 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.200853 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" containerName="manager" containerID="cri-o://843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f" gracePeriod=2 Jul 21 14:26:12.203038 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.202985 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.208022 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.207980 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp"] Jul 21 14:26:12.215458 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.215419 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82"] Jul 21 14:26:12.216209 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.216171 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" containerName="manager" Jul 21 14:26:12.216209 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.216201 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" containerName="manager" Jul 21 14:26:12.216396 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.216302 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" containerName="manager" Jul 21 14:26:12.219440 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.219418 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.220356 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.220329 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k"] Jul 21 14:26:12.220678 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.220540 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" containerName="manager" containerID="cri-o://c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1" gracePeriod=2 Jul 21 14:26:12.223803 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.223769 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.229981 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.229956 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k"] Jul 21 14:26:12.236799 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.236773 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82"] Jul 21 14:26:12.259321 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.259289 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.266108 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.266083 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl"] Jul 21 14:26:12.266443 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.266426 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" containerName="manager" Jul 21 14:26:12.266443 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.266443 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" containerName="manager" Jul 21 14:26:12.266569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.266502 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" containerName="manager" Jul 21 14:26:12.267461 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.267430 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.269338 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.269321 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:12.285934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.285914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl"] Jul 21 14:26:12.307047 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.307002 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.309361 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.309340 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.390608 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.390574 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53541dae-24d5-49a2-83b4-4b9615c62786-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbt82\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.390744 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.390632 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45h77\" (UniqueName: \"kubernetes.io/projected/53541dae-24d5-49a2-83b4-4b9615c62786-kube-api-access-45h77\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbt82\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.390816 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.390741 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgtk\" (UniqueName: \"kubernetes.io/projected/c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc-kube-api-access-9xgtk\") pod \"limitador-operator-controller-manager-859d7b78bd-r7tsl\" (UID: \"c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:12.456639 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.456591 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.458954 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.458928 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.459321 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.459306 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.461001 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.460979 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.463057 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.463038 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.464951 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.464932 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.491173 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.491142 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgtk\" (UniqueName: \"kubernetes.io/projected/c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc-kube-api-access-9xgtk\") pod \"limitador-operator-controller-manager-859d7b78bd-r7tsl\" (UID: \"c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:12.491274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.491212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53541dae-24d5-49a2-83b4-4b9615c62786-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbt82\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.491274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.491247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45h77\" (UniqueName: \"kubernetes.io/projected/53541dae-24d5-49a2-83b4-4b9615c62786-kube-api-access-45h77\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbt82\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.491605 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.491582 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53541dae-24d5-49a2-83b4-4b9615c62786-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbt82\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.492178 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.492157 2570 generic.go:356] "Generic (PLEG): container finished" podID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" containerID="843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f" exitCode=0 Jul 21 14:26:12.492264 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.492201 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.492264 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.492255 2570 scope.go:117] "RemoveContainer" containerID="843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f" Jul 21 14:26:12.493507 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.493481 2570 generic.go:356] "Generic (PLEG): container finished" podID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" containerID="c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1" exitCode=0 Jul 21 14:26:12.493596 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.493526 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.495162 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.495141 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.497207 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.497187 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.499234 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.499211 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.500549 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.500534 2570 scope.go:117] "RemoveContainer" containerID="843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f" Jul 21 14:26:12.500773 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.500754 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f\": container with ID starting with 843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f not found: ID does not exist" containerID="843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f" Jul 21 14:26:12.500854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.500778 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f"} err="failed to get container status \"843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f\": rpc error: code = NotFound desc = could not find container \"843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f\": container with ID starting with 843438cf91e20996b049ab916e18f048464feb1a5d9d26e15fab1bf17d80789f not found: ID does not exist" Jul 21 14:26:12.500854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.500794 2570 scope.go:117] "RemoveContainer" containerID="c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1" Jul 21 14:26:12.501173 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.501151 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.506419 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.506398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45h77\" (UniqueName: \"kubernetes.io/projected/53541dae-24d5-49a2-83b4-4b9615c62786-kube-api-access-45h77\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-fbt82\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.506493 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.506484 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgtk\" (UniqueName: \"kubernetes.io/projected/c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc-kube-api-access-9xgtk\") pod \"limitador-operator-controller-manager-859d7b78bd-r7tsl\" (UID: \"c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:12.509841 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.509826 2570 scope.go:117] "RemoveContainer" containerID="c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1" Jul 21 14:26:12.510125 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.510108 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1\": container with ID starting with c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1 not found: ID does not exist" containerID="c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1" Jul 21 14:26:12.510193 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.510129 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1"} err="failed to get container status \"c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1\": rpc error: code = NotFound desc = could not find container \"c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1\": container with ID starting with c27a2ed0d41a2cc9d954fe93d8ef6567b45796232076b6bb65a82d1eb99e35a1 not found: ID does not exist" Jul 21 14:26:12.592593 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.592566 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g52bc\" (UniqueName: \"kubernetes.io/projected/41ed6ab1-8979-4bf2-be37-1e0f750dfe59-kube-api-access-g52bc\") pod \"41ed6ab1-8979-4bf2-be37-1e0f750dfe59\" (UID: \"41ed6ab1-8979-4bf2-be37-1e0f750dfe59\") " Jul 21 14:26:12.592687 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.592607 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0f35e7-65dd-4b26-878f-2f001df66ae7-extensions-socket-volume\") pod \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " Jul 21 14:26:12.592687 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.592640 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq62x\" (UniqueName: \"kubernetes.io/projected/ce0f35e7-65dd-4b26-878f-2f001df66ae7-kube-api-access-xq62x\") pod \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\" (UID: \"ce0f35e7-65dd-4b26-878f-2f001df66ae7\") " Jul 21 14:26:12.593081 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.593059 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce0f35e7-65dd-4b26-878f-2f001df66ae7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ce0f35e7-65dd-4b26-878f-2f001df66ae7" (UID: "ce0f35e7-65dd-4b26-878f-2f001df66ae7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:26:12.594394 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.594374 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ed6ab1-8979-4bf2-be37-1e0f750dfe59-kube-api-access-g52bc" (OuterVolumeSpecName: "kube-api-access-g52bc") pod "41ed6ab1-8979-4bf2-be37-1e0f750dfe59" (UID: "41ed6ab1-8979-4bf2-be37-1e0f750dfe59"). InnerVolumeSpecName "kube-api-access-g52bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:26:12.594457 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.594418 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0f35e7-65dd-4b26-878f-2f001df66ae7-kube-api-access-xq62x" (OuterVolumeSpecName: "kube-api-access-xq62x") pod "ce0f35e7-65dd-4b26-878f-2f001df66ae7" (UID: "ce0f35e7-65dd-4b26-878f-2f001df66ae7"). InnerVolumeSpecName "kube-api-access-xq62x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:26:12.658614 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.658591 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:12.665295 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.665275 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:12.693366 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.693337 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52bc\" (UniqueName: \"kubernetes.io/projected/41ed6ab1-8979-4bf2-be37-1e0f750dfe59-kube-api-access-g52bc\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:26:12.693366 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.693361 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce0f35e7-65dd-4b26-878f-2f001df66ae7-extensions-socket-volume\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:26:12.693555 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.693373 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq62x\" (UniqueName: \"kubernetes.io/projected/ce0f35e7-65dd-4b26-878f-2f001df66ae7-kube-api-access-xq62x\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:26:12.805083 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.805058 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82"] Jul 21 14:26:12.808659 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.808626 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.808804 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:26:12.808782 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53541dae_24d5_49a2_83b4_4b9615c62786.slice/crio-23fa718871dc7458ed29243dcc9f829eab9b5d4a33eadcefdf27883d4161c374 WatchSource:0}: Error finding container 23fa718871dc7458ed29243dcc9f829eab9b5d4a33eadcefdf27883d4161c374: Status 404 returned error can't find the container with id 23fa718871dc7458ed29243dcc9f829eab9b5d4a33eadcefdf27883d4161c374 Jul 21 14:26:12.810985 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.810966 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.813295 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.813272 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:12.815356 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:12.815334 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:12.824696 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:12.824677 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl"] Jul 21 14:26:12.826405 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:26:12.826382 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f78da9_3ed0_4f90_82d7_c64eeb4f5bcc.slice/crio-e6197c6904f60f91e9865ac240d5b65acc601d334b414842419e6e85494156a8 WatchSource:0}: Error finding container e6197c6904f60f91e9865ac240d5b65acc601d334b414842419e6e85494156a8: Status 404 returned error can't find the container with id e6197c6904f60f91e9865ac240d5b65acc601d334b414842419e6e85494156a8 Jul 21 14:26:13.261448 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.261410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:13.499248 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.499171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" event={"ID":"c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc","Type":"ContainerStarted","Data":"7d4b7b56843e223ff73b0a90bfcc0dbfe9b6ab0cb1caef6077a4dfc85ae2e349"} Jul 21 14:26:13.499248 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.499228 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" event={"ID":"c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc","Type":"ContainerStarted","Data":"e6197c6904f60f91e9865ac240d5b65acc601d334b414842419e6e85494156a8"} Jul 21 14:26:13.499248 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.499249 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:13.500861 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.500836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" event={"ID":"53541dae-24d5-49a2-83b4-4b9615c62786","Type":"ContainerStarted","Data":"1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d"} Jul 21 14:26:13.501019 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.500881 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" event={"ID":"53541dae-24d5-49a2-83b4-4b9615c62786","Type":"ContainerStarted","Data":"23fa718871dc7458ed29243dcc9f829eab9b5d4a33eadcefdf27883d4161c374"} Jul 21 14:26:13.501019 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.500911 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:13.501555 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.501526 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:13.519743 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.519705 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" podStartSLOduration=1.519693643 podStartE2EDuration="1.519693643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:26:13.517891923 +0000 UTC m=+535.885401951" watchObservedRunningTime="2026-07-21 14:26:13.519693643 +0000 UTC m=+535.887203670" Jul 21 14:26:13.520068 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:13.520047 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:13.522303 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:13.522283 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:13.524289 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:13.524271 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5fk9k\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5fk9k" Jul 21 14:26:13.526346 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:13.526324 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qlzbp\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qlzbp" Jul 21 14:26:13.545100 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:13.545059 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" podStartSLOduration=1.545044464 podStartE2EDuration="1.545044464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:26:13.544322711 +0000 UTC m=+535.911832738" watchObservedRunningTime="2026-07-21 14:26:13.545044464 +0000 UTC m=+535.912554489" Jul 21 14:26:14.141496 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:14.141451 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ed6ab1-8979-4bf2-be37-1e0f750dfe59" path="/var/lib/kubelet/pods/41ed6ab1-8979-4bf2-be37-1e0f750dfe59/volumes" Jul 21 14:26:14.141814 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:14.141797 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce0f35e7-65dd-4b26-878f-2f001df66ae7" path="/var/lib/kubelet/pods/ce0f35e7-65dd-4b26-878f-2f001df66ae7/volumes" Jul 21 14:26:16.495082 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:16.495039 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:16.596950 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:16.596914 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:16.906816 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:16.906778 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:17.007185 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:17.007156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:20.141231 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:20.141192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:20.253805 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:20.253770 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:24.508885 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.508848 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r7tsl" Jul 21 14:26:24.509332 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.508955 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:24.567822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.567787 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:24.646148 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.646106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:24.796649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.796580 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf"] Jul 21 14:26:24.803070 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.803049 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:24.813533 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.813512 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf"] Jul 21 14:26:24.887407 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.887373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/542fd1e5-2c0f-4377-a994-75f049860445-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6vwzf\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:24.887546 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.887426 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6kz\" (UniqueName: \"kubernetes.io/projected/542fd1e5-2c0f-4377-a994-75f049860445-kube-api-access-hg6kz\") pod \"kuadrant-operator-controller-manager-86878c769f-6vwzf\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:24.988171 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.988147 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/542fd1e5-2c0f-4377-a994-75f049860445-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6vwzf\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:24.988282 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.988194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6kz\" (UniqueName: \"kubernetes.io/projected/542fd1e5-2c0f-4377-a994-75f049860445-kube-api-access-hg6kz\") pod \"kuadrant-operator-controller-manager-86878c769f-6vwzf\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:24.988498 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:24.988479 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/542fd1e5-2c0f-4377-a994-75f049860445-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6vwzf\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:25.005444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.005425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6kz\" (UniqueName: \"kubernetes.io/projected/542fd1e5-2c0f-4377-a994-75f049860445-kube-api-access-hg6kz\") pod \"kuadrant-operator-controller-manager-86878c769f-6vwzf\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:25.113643 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.113579 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:25.260124 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.260093 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf"] Jul 21 14:26:25.263578 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:26:25.263549 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542fd1e5_2c0f_4377_a994_75f049860445.slice/crio-cdcae0e9c6c90b509fb2ee4d0c6e86ccd44b4dff6e90ea90067a51191b0dab5e WatchSource:0}: Error finding container cdcae0e9c6c90b509fb2ee4d0c6e86ccd44b4dff6e90ea90067a51191b0dab5e: Status 404 returned error can't find the container with id cdcae0e9c6c90b509fb2ee4d0c6e86ccd44b4dff6e90ea90067a51191b0dab5e Jul 21 14:26:25.563851 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.563818 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" event={"ID":"542fd1e5-2c0f-4377-a994-75f049860445","Type":"ContainerStarted","Data":"6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa"} Jul 21 14:26:25.564347 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.563866 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" event={"ID":"542fd1e5-2c0f-4377-a994-75f049860445","Type":"ContainerStarted","Data":"cdcae0e9c6c90b509fb2ee4d0c6e86ccd44b4dff6e90ea90067a51191b0dab5e"} Jul 21 14:26:25.564347 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.563930 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:25.592516 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:25.592477 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" podStartSLOduration=1.592466387 podStartE2EDuration="1.592466387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:26:25.5909809 +0000 UTC m=+547.958490928" watchObservedRunningTime="2026-07-21 14:26:25.592466387 +0000 UTC m=+547.959976405" Jul 21 14:26:26.353387 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:26.353355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:27.941524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:27.941454 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:28.045837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:28.045807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:36.571658 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.571601 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:26:36.637075 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.637043 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82"] Jul 21 14:26:36.637252 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.637216 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" podUID="53541dae-24d5-49a2-83b4-4b9615c62786" containerName="manager" containerID="cri-o://1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d" gracePeriod=10 Jul 21 14:26:36.874262 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.874241 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:36.974159 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.974135 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53541dae-24d5-49a2-83b4-4b9615c62786-extensions-socket-volume\") pod \"53541dae-24d5-49a2-83b4-4b9615c62786\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " Jul 21 14:26:36.974268 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.974234 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45h77\" (UniqueName: \"kubernetes.io/projected/53541dae-24d5-49a2-83b4-4b9615c62786-kube-api-access-45h77\") pod \"53541dae-24d5-49a2-83b4-4b9615c62786\" (UID: \"53541dae-24d5-49a2-83b4-4b9615c62786\") " Jul 21 14:26:36.974469 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.974447 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53541dae-24d5-49a2-83b4-4b9615c62786-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53541dae-24d5-49a2-83b4-4b9615c62786" (UID: "53541dae-24d5-49a2-83b4-4b9615c62786"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:26:36.976117 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:36.976092 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53541dae-24d5-49a2-83b4-4b9615c62786-kube-api-access-45h77" (OuterVolumeSpecName: "kube-api-access-45h77") pod "53541dae-24d5-49a2-83b4-4b9615c62786" (UID: "53541dae-24d5-49a2-83b4-4b9615c62786"). InnerVolumeSpecName "kube-api-access-45h77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:26:37.074656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.074637 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53541dae-24d5-49a2-83b4-4b9615c62786-extensions-socket-volume\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:26:37.074656 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.074657 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45h77\" (UniqueName: \"kubernetes.io/projected/53541dae-24d5-49a2-83b4-4b9615c62786-kube-api-access-45h77\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:26:37.621603 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.621568 2570 generic.go:356] "Generic (PLEG): container finished" podID="53541dae-24d5-49a2-83b4-4b9615c62786" containerID="1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d" exitCode=0 Jul 21 14:26:37.621957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.621613 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" Jul 21 14:26:37.621957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.621621 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" event={"ID":"53541dae-24d5-49a2-83b4-4b9615c62786","Type":"ContainerDied","Data":"1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d"} Jul 21 14:26:37.621957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.621675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82" event={"ID":"53541dae-24d5-49a2-83b4-4b9615c62786","Type":"ContainerDied","Data":"23fa718871dc7458ed29243dcc9f829eab9b5d4a33eadcefdf27883d4161c374"} Jul 21 14:26:37.621957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.621700 2570 scope.go:117] "RemoveContainer" containerID="1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d" Jul 21 14:26:37.631429 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.631411 2570 scope.go:117] "RemoveContainer" containerID="1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d" Jul 21 14:26:37.631725 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:26:37.631705 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d\": container with ID starting with 1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d not found: ID does not exist" containerID="1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d" Jul 21 14:26:37.631794 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.631734 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d"} err="failed to get container status \"1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d\": rpc error: code = NotFound desc = could not find container \"1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d\": container with ID starting with 1ebcdadd40c23fb4559cb41219ac5dc7a1ac99f2e5f6fb624a60db3d57fb339d not found: ID does not exist" Jul 21 14:26:37.646716 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.646690 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82"] Jul 21 14:26:37.651500 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:37.651477 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-fbt82"] Jul 21 14:26:38.141244 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:38.141216 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53541dae-24d5-49a2-83b4-4b9615c62786" path="/var/lib/kubelet/pods/53541dae-24d5-49a2-83b4-4b9615c62786/volumes" Jul 21 14:26:39.004346 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:39.004312 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:39.667470 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:39.667442 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:39.769161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:39.769135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:41.019320 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:41.019278 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:41.216562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:41.216533 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:56.460431 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.460393 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:56.461700 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.461678 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:26:56.462040 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.462027 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53541dae-24d5-49a2-83b4-4b9615c62786" containerName="manager" Jul 21 14:26:56.462082 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.462041 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="53541dae-24d5-49a2-83b4-4b9615c62786" containerName="manager" Jul 21 14:26:56.462115 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.462102 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="53541dae-24d5-49a2-83b4-4b9615c62786" containerName="manager" Jul 21 14:26:56.464096 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.464080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.466593 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.466571 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q6lsw\"" Jul 21 14:26:56.466707 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.466614 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 14:26:56.472620 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.472599 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:26:56.522752 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.522726 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8p8\" (UniqueName: \"kubernetes.io/projected/37910954-7493-44ec-a583-1123a09e0413-kube-api-access-bq8p8\") pod \"limitador-limitador-64c94fd64f-q2srg\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.522864 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.522766 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37910954-7493-44ec-a583-1123a09e0413-config-file\") pod \"limitador-limitador-64c94fd64f-q2srg\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.552709 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.552684 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:26:56.559163 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.559137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:56.623945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.623924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8p8\" (UniqueName: \"kubernetes.io/projected/37910954-7493-44ec-a583-1123a09e0413-kube-api-access-bq8p8\") pod \"limitador-limitador-64c94fd64f-q2srg\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.624065 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.623957 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37910954-7493-44ec-a583-1123a09e0413-config-file\") pod \"limitador-limitador-64c94fd64f-q2srg\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.624488 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.624470 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37910954-7493-44ec-a583-1123a09e0413-config-file\") pod \"limitador-limitador-64c94fd64f-q2srg\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.625806 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.625785 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:56.632989 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.632973 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8p8\" (UniqueName: \"kubernetes.io/projected/37910954-7493-44ec-a583-1123a09e0413-kube-api-access-bq8p8\") pod \"limitador-limitador-64c94fd64f-q2srg\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.774095 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.773992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:26:56.905268 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:56.905241 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:26:56.908731 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:26:56.908700 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37910954_7493_44ec_a583_1123a09e0413.slice/crio-8367b6a02d56a2162096736414167e506814ea9c13bb0fcb9c93410d3e53fc06 WatchSource:0}: Error finding container 8367b6a02d56a2162096736414167e506814ea9c13bb0fcb9c93410d3e53fc06: Status 404 returned error can't find the container with id 8367b6a02d56a2162096736414167e506814ea9c13bb0fcb9c93410d3e53fc06 Jul 21 14:26:57.341161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.341132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:57.423158 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.423134 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:26:57.449474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.449440 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-d6hz9"] Jul 21 14:26:57.452444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.452425 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:26:57.455084 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.455064 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-596n2\"" Jul 21 14:26:57.460558 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.460539 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d6hz9"] Jul 21 14:26:57.532623 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.532600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dz5l\" (UniqueName: \"kubernetes.io/projected/31966786-ee81-4b4e-af0e-215a7309b612-kube-api-access-2dz5l\") pod \"authorino-75d556bd77-d6hz9\" (UID: \"31966786-ee81-4b4e-af0e-215a7309b612\") " pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:26:57.633313 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.633259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dz5l\" (UniqueName: \"kubernetes.io/projected/31966786-ee81-4b4e-af0e-215a7309b612-kube-api-access-2dz5l\") pod \"authorino-75d556bd77-d6hz9\" (UID: \"31966786-ee81-4b4e-af0e-215a7309b612\") " pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:26:57.642352 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.642324 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dz5l\" (UniqueName: \"kubernetes.io/projected/31966786-ee81-4b4e-af0e-215a7309b612-kube-api-access-2dz5l\") pod \"authorino-75d556bd77-d6hz9\" (UID: \"31966786-ee81-4b4e-af0e-215a7309b612\") " pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:26:57.723189 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.723165 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" event={"ID":"37910954-7493-44ec-a583-1123a09e0413","Type":"ContainerStarted","Data":"8367b6a02d56a2162096736414167e506814ea9c13bb0fcb9c93410d3e53fc06"} Jul 21 14:26:57.762619 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.762598 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:26:57.912313 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:57.912282 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d6hz9"] Jul 21 14:26:57.918327 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:26:57.918292 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31966786_ee81_4b4e_af0e_215a7309b612.slice/crio-849aa99bc9604fcfa0c6f0021a9d6121c0e58a33fba5f1c5c810fd53d75f3e34 WatchSource:0}: Error finding container 849aa99bc9604fcfa0c6f0021a9d6121c0e58a33fba5f1c5c810fd53d75f3e34: Status 404 returned error can't find the container with id 849aa99bc9604fcfa0c6f0021a9d6121c0e58a33fba5f1c5c810fd53d75f3e34 Jul 21 14:26:58.731319 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:26:58.731273 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d6hz9" event={"ID":"31966786-ee81-4b4e-af0e-215a7309b612","Type":"ContainerStarted","Data":"849aa99bc9604fcfa0c6f0021a9d6121c0e58a33fba5f1c5c810fd53d75f3e34"} Jul 21 14:27:01.754599 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:01.754561 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" event={"ID":"37910954-7493-44ec-a583-1123a09e0413","Type":"ContainerStarted","Data":"995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076"} Jul 21 14:27:01.755033 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:01.754719 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:27:01.756059 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:01.756029 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d6hz9" event={"ID":"31966786-ee81-4b4e-af0e-215a7309b612","Type":"ContainerStarted","Data":"a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9"} Jul 21 14:27:01.775789 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:01.775743 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" podStartSLOduration=1.106542922 podStartE2EDuration="5.77573014s" totalImagesPullingTime="4.669187218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:27:01.773779781 +0000 UTC m=+584.141289810" watchObservedRunningTime="2026-07-21 14:27:01.77573014 +0000 UTC m=+584.143240212" Jul 21 14:27:01.790871 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:01.790833 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-d6hz9" podStartSLOduration=1.124792284 podStartE2EDuration="4.790821488s" totalImagesPullingTime="3.666029204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:27:01.79041933 +0000 UTC m=+584.157929358" watchObservedRunningTime="2026-07-21 14:27:01.790821488 +0000 UTC m=+584.158331515" Jul 21 14:27:12.761840 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:12.761808 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:27:13.675551 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:13.675506 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:27:13.675808 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:13.675782 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" podUID="37910954-7493-44ec-a583-1123a09e0413" containerName="limitador" containerID="cri-o://995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076" gracePeriod=30 Jul 21 14:27:14.216517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.216495 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:27:14.364128 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.364069 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq8p8\" (UniqueName: \"kubernetes.io/projected/37910954-7493-44ec-a583-1123a09e0413-kube-api-access-bq8p8\") pod \"37910954-7493-44ec-a583-1123a09e0413\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " Jul 21 14:27:14.364128 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.364120 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37910954-7493-44ec-a583-1123a09e0413-config-file\") pod \"37910954-7493-44ec-a583-1123a09e0413\" (UID: \"37910954-7493-44ec-a583-1123a09e0413\") " Jul 21 14:27:14.364452 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.364431 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37910954-7493-44ec-a583-1123a09e0413-config-file" (OuterVolumeSpecName: "config-file") pod "37910954-7493-44ec-a583-1123a09e0413" (UID: "37910954-7493-44ec-a583-1123a09e0413"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:27:14.365967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.365944 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37910954-7493-44ec-a583-1123a09e0413-kube-api-access-bq8p8" (OuterVolumeSpecName: "kube-api-access-bq8p8") pod "37910954-7493-44ec-a583-1123a09e0413" (UID: "37910954-7493-44ec-a583-1123a09e0413"). InnerVolumeSpecName "kube-api-access-bq8p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:27:14.464628 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.464607 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq8p8\" (UniqueName: \"kubernetes.io/projected/37910954-7493-44ec-a583-1123a09e0413-kube-api-access-bq8p8\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:27:14.464628 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.464628 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/37910954-7493-44ec-a583-1123a09e0413-config-file\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:27:14.817815 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.817782 2570 generic.go:356] "Generic (PLEG): container finished" podID="37910954-7493-44ec-a583-1123a09e0413" containerID="995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076" exitCode=0 Jul 21 14:27:14.817921 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.817827 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" Jul 21 14:27:14.817921 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.817833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" event={"ID":"37910954-7493-44ec-a583-1123a09e0413","Type":"ContainerDied","Data":"995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076"} Jul 21 14:27:14.817921 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.817886 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-q2srg" event={"ID":"37910954-7493-44ec-a583-1123a09e0413","Type":"ContainerDied","Data":"8367b6a02d56a2162096736414167e506814ea9c13bb0fcb9c93410d3e53fc06"} Jul 21 14:27:14.817921 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.817916 2570 scope.go:117] "RemoveContainer" containerID="995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076" Jul 21 14:27:14.826747 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.826725 2570 scope.go:117] "RemoveContainer" containerID="995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076" Jul 21 14:27:14.827039 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:27:14.827000 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076\": container with ID starting with 995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076 not found: ID does not exist" containerID="995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076" Jul 21 14:27:14.827113 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.827054 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076"} err="failed to get container status \"995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076\": rpc error: code = NotFound desc = could not find container \"995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076\": container with ID starting with 995437f1da379a3c481d63fc015d620d20870b5be7e1607dc5cebcadafb46076 not found: ID does not exist" Jul 21 14:27:14.840715 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.840693 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:27:14.845339 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:14.845317 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-q2srg"] Jul 21 14:27:15.188205 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.188175 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:15.287117 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.287089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:15.661560 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.661529 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl"] Jul 21 14:27:15.661954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.661938 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37910954-7493-44ec-a583-1123a09e0413" containerName="limitador" Jul 21 14:27:15.661954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.661954 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="37910954-7493-44ec-a583-1123a09e0413" containerName="limitador" Jul 21 14:27:15.662118 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.662055 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="37910954-7493-44ec-a583-1123a09e0413" containerName="limitador" Jul 21 14:27:15.666253 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.666227 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.668748 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.668723 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 14:27:15.669835 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.669806 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 14:27:15.669974 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.669879 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-qwdcz\"" Jul 21 14:27:15.669974 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.669888 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 14:27:15.672931 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.672906 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl"] Jul 21 14:27:15.696766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.696740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:15.775837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.775811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40624fc1-938e-4096-b24a-0e57bd4de01f-data\") pod \"postgres-69fb4594cf-vdcnl\" (UID: \"40624fc1-938e-4096-b24a-0e57bd4de01f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.775935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.775870 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smvb\" (UniqueName: \"kubernetes.io/projected/40624fc1-938e-4096-b24a-0e57bd4de01f-kube-api-access-6smvb\") pod \"postgres-69fb4594cf-vdcnl\" (UID: \"40624fc1-938e-4096-b24a-0e57bd4de01f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.876739 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.876712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smvb\" (UniqueName: \"kubernetes.io/projected/40624fc1-938e-4096-b24a-0e57bd4de01f-kube-api-access-6smvb\") pod \"postgres-69fb4594cf-vdcnl\" (UID: \"40624fc1-938e-4096-b24a-0e57bd4de01f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.876850 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.876777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40624fc1-938e-4096-b24a-0e57bd4de01f-data\") pod \"postgres-69fb4594cf-vdcnl\" (UID: \"40624fc1-938e-4096-b24a-0e57bd4de01f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.877102 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.877086 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40624fc1-938e-4096-b24a-0e57bd4de01f-data\") pod \"postgres-69fb4594cf-vdcnl\" (UID: \"40624fc1-938e-4096-b24a-0e57bd4de01f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.888184 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.888165 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smvb\" (UniqueName: \"kubernetes.io/projected/40624fc1-938e-4096-b24a-0e57bd4de01f-kube-api-access-6smvb\") pod \"postgres-69fb4594cf-vdcnl\" (UID: \"40624fc1-938e-4096-b24a-0e57bd4de01f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:15.979347 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:15.979298 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:16.107418 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:16.107396 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl"] Jul 21 14:27:16.110211 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:27:16.110187 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40624fc1_938e_4096_b24a_0e57bd4de01f.slice/crio-ad9c3d5eb4b05a33536601d68d4f36cba836ae34c9160113fc342c3a4254108a WatchSource:0}: Error finding container ad9c3d5eb4b05a33536601d68d4f36cba836ae34c9160113fc342c3a4254108a: Status 404 returned error can't find the container with id ad9c3d5eb4b05a33536601d68d4f36cba836ae34c9160113fc342c3a4254108a Jul 21 14:27:16.141117 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:16.141097 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37910954-7493-44ec-a583-1123a09e0413" path="/var/lib/kubelet/pods/37910954-7493-44ec-a583-1123a09e0413/volumes" Jul 21 14:27:16.832070 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:16.832039 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" event={"ID":"40624fc1-938e-4096-b24a-0e57bd4de01f","Type":"ContainerStarted","Data":"ad9c3d5eb4b05a33536601d68d4f36cba836ae34c9160113fc342c3a4254108a"} Jul 21 14:27:18.081459 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:18.081429 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:27:18.082404 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:18.082374 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:27:20.901828 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:20.901809 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 14:27:21.858229 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:21.858189 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" event={"ID":"40624fc1-938e-4096-b24a-0e57bd4de01f","Type":"ContainerStarted","Data":"98cd5338d5a5d44302368c7f4e25776bde57661007ffc1c882c602f89250d96b"} Jul 21 14:27:21.858436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:21.858292 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:21.877567 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:21.877520 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" podStartSLOduration=2.090476217 podStartE2EDuration="6.877504762s" totalImagesPullingTime="4.787028545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:27:21.877087277 +0000 UTC m=+604.244597305" watchObservedRunningTime="2026-07-21 14:27:21.877504762 +0000 UTC m=+604.245014821" Jul 21 14:27:27.893669 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:27.893640 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-vdcnl" Jul 21 14:27:28.268003 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.267890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:28.397660 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.397623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:28.766214 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.766186 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76d445665-z2j9d"] Jul 21 14:27:28.771960 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.771942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:28.778149 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.778131 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d445665-z2j9d"] Jul 21 14:27:28.865100 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.865081 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d445665-z2j9d"] Jul 21 14:27:28.865300 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:27:28.865280 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dlhxh], unattached volumes=[], failed to process volumes=[kube-api-access-dlhxh]: context canceled" pod="kuadrant-system/authorino-76d445665-z2j9d" podUID="45df7344-408c-4584-a28d-ce5ef7f3e411" Jul 21 14:27:28.876834 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.876816 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhxh\" (UniqueName: \"kubernetes.io/projected/45df7344-408c-4584-a28d-ce5ef7f3e411-kube-api-access-dlhxh\") pod \"authorino-76d445665-z2j9d\" (UID: \"45df7344-408c-4584-a28d-ce5ef7f3e411\") " pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:28.896391 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.896368 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:28.903283 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.903266 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:28.977668 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.977639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhxh\" (UniqueName: \"kubernetes.io/projected/45df7344-408c-4584-a28d-ce5ef7f3e411-kube-api-access-dlhxh\") pod \"authorino-76d445665-z2j9d\" (UID: \"45df7344-408c-4584-a28d-ce5ef7f3e411\") " pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:28.991960 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:28.991940 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhxh\" (UniqueName: \"kubernetes.io/projected/45df7344-408c-4584-a28d-ce5ef7f3e411-kube-api-access-dlhxh\") pod \"authorino-76d445665-z2j9d\" (UID: \"45df7344-408c-4584-a28d-ce5ef7f3e411\") " pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:29.078481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:29.078428 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlhxh\" (UniqueName: \"kubernetes.io/projected/45df7344-408c-4584-a28d-ce5ef7f3e411-kube-api-access-dlhxh\") pod \"45df7344-408c-4584-a28d-ce5ef7f3e411\" (UID: \"45df7344-408c-4584-a28d-ce5ef7f3e411\") " Jul 21 14:27:29.080324 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:29.080297 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45df7344-408c-4584-a28d-ce5ef7f3e411-kube-api-access-dlhxh" (OuterVolumeSpecName: "kube-api-access-dlhxh") pod "45df7344-408c-4584-a28d-ce5ef7f3e411" (UID: "45df7344-408c-4584-a28d-ce5ef7f3e411"). InnerVolumeSpecName "kube-api-access-dlhxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:27:29.179522 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:29.179500 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhxh\" (UniqueName: \"kubernetes.io/projected/45df7344-408c-4584-a28d-ce5ef7f3e411-kube-api-access-dlhxh\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:27:29.901733 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:29.901700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d445665-z2j9d" Jul 21 14:27:29.938530 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:29.938492 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d445665-z2j9d"] Jul 21 14:27:29.942207 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:29.942181 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76d445665-z2j9d"] Jul 21 14:27:30.141399 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:30.141368 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45df7344-408c-4584-a28d-ce5ef7f3e411" path="/var/lib/kubelet/pods/45df7344-408c-4584-a28d-ce5ef7f3e411/volumes" Jul 21 14:27:30.793749 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:30.793712 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d6hz9"] Jul 21 14:27:30.793916 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:30.793875 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-d6hz9" podUID="31966786-ee81-4b4e-af0e-215a7309b612" containerName="authorino" containerID="cri-o://a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9" gracePeriod=30 Jul 21 14:27:31.074210 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.074185 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:27:31.094444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.094425 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dz5l\" (UniqueName: \"kubernetes.io/projected/31966786-ee81-4b4e-af0e-215a7309b612-kube-api-access-2dz5l\") pod \"31966786-ee81-4b4e-af0e-215a7309b612\" (UID: \"31966786-ee81-4b4e-af0e-215a7309b612\") " Jul 21 14:27:31.096492 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.096465 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31966786-ee81-4b4e-af0e-215a7309b612-kube-api-access-2dz5l" (OuterVolumeSpecName: "kube-api-access-2dz5l") pod "31966786-ee81-4b4e-af0e-215a7309b612" (UID: "31966786-ee81-4b4e-af0e-215a7309b612"). InnerVolumeSpecName "kube-api-access-2dz5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:27:31.195982 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.195955 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dz5l\" (UniqueName: \"kubernetes.io/projected/31966786-ee81-4b4e-af0e-215a7309b612-kube-api-access-2dz5l\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:27:31.911176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.911142 2570 generic.go:356] "Generic (PLEG): container finished" podID="31966786-ee81-4b4e-af0e-215a7309b612" containerID="a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9" exitCode=0 Jul 21 14:27:31.911319 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.911183 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-d6hz9" Jul 21 14:27:31.911319 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.911205 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d6hz9" event={"ID":"31966786-ee81-4b4e-af0e-215a7309b612","Type":"ContainerDied","Data":"a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9"} Jul 21 14:27:31.911319 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.911253 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-d6hz9" event={"ID":"31966786-ee81-4b4e-af0e-215a7309b612","Type":"ContainerDied","Data":"849aa99bc9604fcfa0c6f0021a9d6121c0e58a33fba5f1c5c810fd53d75f3e34"} Jul 21 14:27:31.911319 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.911278 2570 scope.go:117] "RemoveContainer" containerID="a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9" Jul 21 14:27:31.919701 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.919680 2570 scope.go:117] "RemoveContainer" containerID="a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9" Jul 21 14:27:31.919948 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:27:31.919932 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9\": container with ID starting with a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9 not found: ID does not exist" containerID="a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9" Jul 21 14:27:31.920032 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.919955 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9"} err="failed to get container status \"a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9\": rpc error: code = NotFound desc = could not find container \"a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9\": container with ID starting with a61bd39ac7064c39958dcac54b770f204c0015a407e3c338e3e94d20701d9ba9 not found: ID does not exist" Jul 21 14:27:31.933773 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.933749 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d6hz9"] Jul 21 14:27:31.938777 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:31.938755 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-d6hz9"] Jul 21 14:27:32.141228 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:32.141205 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31966786-ee81-4b4e-af0e-215a7309b612" path="/var/lib/kubelet/pods/31966786-ee81-4b4e-af0e-215a7309b612/volumes" Jul 21 14:27:35.778403 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.778364 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:35.779243 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.779221 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d44b6c5c4-hzmt7"] Jul 21 14:27:35.779618 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.779604 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31966786-ee81-4b4e-af0e-215a7309b612" containerName="authorino" Jul 21 14:27:35.779670 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.779620 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="31966786-ee81-4b4e-af0e-215a7309b612" containerName="authorino" Jul 21 14:27:35.779713 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.779681 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="31966786-ee81-4b4e-af0e-215a7309b612" containerName="authorino" Jul 21 14:27:35.860590 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.860561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:35.883701 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.883677 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d44b6c5c4-hzmt7"] Jul 21 14:27:35.883791 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.883776 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:35.887490 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.887466 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 14:27:35.887613 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.887512 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cpwp7\"" Jul 21 14:27:35.887613 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.887532 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 14:27:35.931264 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.931244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-webhook-cert\") pod \"maas-controller-5d44b6c5c4-hzmt7\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:35.931357 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.931282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkntn\" (UniqueName: \"kubernetes.io/projected/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-kube-api-access-kkntn\") pod \"maas-controller-5d44b6c5c4-hzmt7\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:35.940047 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.940024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:35.970409 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.970388 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b95d7b79-pf4kt"] Jul 21 14:27:35.972862 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.972846 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:35.985252 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:35.985233 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-pf4kt"] Jul 21 14:27:36.031884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.031837 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-webhook-cert\") pod \"maas-controller-5d44b6c5c4-hzmt7\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:36.031884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.031864 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8sf\" (UniqueName: \"kubernetes.io/projected/3f90491d-510a-45e6-98b1-db54dddffed5-kube-api-access-zn8sf\") pod \"maas-controller-84b95d7b79-pf4kt\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.031884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.031888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f90491d-510a-45e6-98b1-db54dddffed5-webhook-cert\") pod \"maas-controller-84b95d7b79-pf4kt\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.032084 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.031923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkntn\" (UniqueName: \"kubernetes.io/projected/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-kube-api-access-kkntn\") pod \"maas-controller-5d44b6c5c4-hzmt7\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:36.034258 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.034234 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-webhook-cert\") pod \"maas-controller-5d44b6c5c4-hzmt7\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:36.040894 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.040875 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkntn\" (UniqueName: \"kubernetes.io/projected/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-kube-api-access-kkntn\") pod \"maas-controller-5d44b6c5c4-hzmt7\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:36.132462 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.132440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f90491d-510a-45e6-98b1-db54dddffed5-webhook-cert\") pod \"maas-controller-84b95d7b79-pf4kt\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.132564 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.132510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8sf\" (UniqueName: \"kubernetes.io/projected/3f90491d-510a-45e6-98b1-db54dddffed5-kube-api-access-zn8sf\") pod \"maas-controller-84b95d7b79-pf4kt\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.134466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.134444 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f90491d-510a-45e6-98b1-db54dddffed5-webhook-cert\") pod \"maas-controller-84b95d7b79-pf4kt\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.141299 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.141280 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8sf\" (UniqueName: \"kubernetes.io/projected/3f90491d-510a-45e6-98b1-db54dddffed5-kube-api-access-zn8sf\") pod \"maas-controller-84b95d7b79-pf4kt\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.193533 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.193511 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:36.282352 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.282277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:36.321692 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.321669 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d44b6c5c4-hzmt7"] Jul 21 14:27:36.324764 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:27:36.324726 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb589cbb8_6808_4d9b_b4d2_827e3b2b22d0.slice/crio-49606a18f8aa21abc5a81c84e0cca8cb3b9d70c0a4292b95a75e5e926667dc97 WatchSource:0}: Error finding container 49606a18f8aa21abc5a81c84e0cca8cb3b9d70c0a4292b95a75e5e926667dc97: Status 404 returned error can't find the container with id 49606a18f8aa21abc5a81c84e0cca8cb3b9d70c0a4292b95a75e5e926667dc97 Jul 21 14:27:36.414464 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.414442 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-pf4kt"] Jul 21 14:27:36.416085 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:27:36.416058 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f90491d_510a_45e6_98b1_db54dddffed5.slice/crio-7aa9bccc44e4f3169bd65bc45562e4d3f3df3aa796728b7c749086a9c709d1b8 WatchSource:0}: Error finding container 7aa9bccc44e4f3169bd65bc45562e4d3f3df3aa796728b7c749086a9c709d1b8: Status 404 returned error can't find the container with id 7aa9bccc44e4f3169bd65bc45562e4d3f3df3aa796728b7c749086a9c709d1b8 Jul 21 14:27:36.941086 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.941053 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" event={"ID":"3f90491d-510a-45e6-98b1-db54dddffed5","Type":"ContainerStarted","Data":"7aa9bccc44e4f3169bd65bc45562e4d3f3df3aa796728b7c749086a9c709d1b8"} Jul 21 14:27:36.942299 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:36.942271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" event={"ID":"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0","Type":"ContainerStarted","Data":"49606a18f8aa21abc5a81c84e0cca8cb3b9d70c0a4292b95a75e5e926667dc97"} Jul 21 14:27:38.744406 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.744372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:38.844849 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.844779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:38.953887 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.953851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" event={"ID":"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0","Type":"ContainerStarted","Data":"b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327"} Jul 21 14:27:38.954063 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.953925 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:38.955345 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.955321 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" event={"ID":"3f90491d-510a-45e6-98b1-db54dddffed5","Type":"ContainerStarted","Data":"89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373"} Jul 21 14:27:38.955465 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.955452 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:38.982641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:38.982600 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" podStartSLOduration=1.5658143789999999 podStartE2EDuration="3.982588005s" totalImagesPullingTime="2.416773626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:27:38.980399615 +0000 UTC m=+621.347909644" watchObservedRunningTime="2026-07-21 14:27:38.982588005 +0000 UTC m=+621.350098033" Jul 21 14:27:39.004841 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:39.004799 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" podStartSLOduration=1.8367180159999998 podStartE2EDuration="4.004787953s" totalImagesPullingTime="2.168069937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:27:39.003433645 +0000 UTC m=+621.370943674" watchObservedRunningTime="2026-07-21 14:27:39.004787953 +0000 UTC m=+621.372297980" Jul 21 14:27:42.019246 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:42.019172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:27:49.962211 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:49.962176 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:27:49.964789 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:49.962596 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:50.047055 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.046997 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d44b6c5c4-hzmt7"] Jul 21 14:27:50.047221 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.047200 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" podUID="b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" containerName="manager" containerID="cri-o://b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327" gracePeriod=30 Jul 21 14:27:50.296660 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.296638 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:50.338490 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.338460 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-webhook-cert\") pod \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " Jul 21 14:27:50.338619 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.338522 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkntn\" (UniqueName: \"kubernetes.io/projected/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-kube-api-access-kkntn\") pod \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\" (UID: \"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0\") " Jul 21 14:27:50.340395 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.340370 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" (UID: "b589cbb8-6808-4d9b-b4d2-827e3b2b22d0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:27:50.340494 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.340467 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-kube-api-access-kkntn" (OuterVolumeSpecName: "kube-api-access-kkntn") pod "b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" (UID: "b589cbb8-6808-4d9b-b4d2-827e3b2b22d0"). InnerVolumeSpecName "kube-api-access-kkntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:27:50.438971 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.438948 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-webhook-cert\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:27:50.438971 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:50.438968 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkntn\" (UniqueName: \"kubernetes.io/projected/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0-kube-api-access-kkntn\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:27:51.015694 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.015658 2570 generic.go:356] "Generic (PLEG): container finished" podID="b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" containerID="b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327" exitCode=0 Jul 21 14:27:51.016043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.015709 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" Jul 21 14:27:51.016043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.015728 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" event={"ID":"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0","Type":"ContainerDied","Data":"b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327"} Jul 21 14:27:51.016043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.015816 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d44b6c5c4-hzmt7" event={"ID":"b589cbb8-6808-4d9b-b4d2-827e3b2b22d0","Type":"ContainerDied","Data":"49606a18f8aa21abc5a81c84e0cca8cb3b9d70c0a4292b95a75e5e926667dc97"} Jul 21 14:27:51.016043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.015848 2570 scope.go:117] "RemoveContainer" containerID="b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327" Jul 21 14:27:51.025651 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.025629 2570 scope.go:117] "RemoveContainer" containerID="b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327" Jul 21 14:27:51.025931 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:27:51.025912 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327\": container with ID starting with b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327 not found: ID does not exist" containerID="b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327" Jul 21 14:27:51.025986 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.025939 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327"} err="failed to get container status \"b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327\": rpc error: code = NotFound desc = could not find container \"b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327\": container with ID starting with b9f3ed20eae914d12ecb9dec360fcba40488abb841ae77c76fe06e466b005327 not found: ID does not exist" Jul 21 14:27:51.039088 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.039058 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d44b6c5c4-hzmt7"] Jul 21 14:27:51.043455 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:51.043425 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d44b6c5c4-hzmt7"] Jul 21 14:27:52.141854 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:27:52.141817 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" path="/var/lib/kubelet/pods/b589cbb8-6808-4d9b-b4d2-827e3b2b22d0/volumes" Jul 21 14:28:09.547747 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:09.547709 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:28:09.994564 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:09.994534 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85cb597c75-ld6th"] Jul 21 14:28:09.994876 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:09.994865 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" containerName="manager" Jul 21 14:28:09.994919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:09.994878 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" containerName="manager" Jul 21 14:28:09.995020 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:09.994998 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b589cbb8-6808-4d9b-b4d2-827e3b2b22d0" containerName="manager" Jul 21 14:28:09.999413 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:09.999394 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.002957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.002935 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 14:28:10.003098 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.002936 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bm4cc\"" Jul 21 14:28:10.009219 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.009193 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cb597c75-ld6th"] Jul 21 14:28:10.082666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.082646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhhx\" (UniqueName: \"kubernetes.io/projected/c3981ae9-d510-40e8-b056-9c3f963afbf4-kube-api-access-bfhhx\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.082780 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.082694 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3981ae9-d510-40e8-b056-9c3f963afbf4-maas-api-tls\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.082780 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.082717 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3981ae9-d510-40e8-b056-9c3f963afbf4-openshift-service-ca\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.183695 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.183674 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3981ae9-d510-40e8-b056-9c3f963afbf4-maas-api-tls\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.183812 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.183710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3981ae9-d510-40e8-b056-9c3f963afbf4-openshift-service-ca\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.183812 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.183743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhhx\" (UniqueName: \"kubernetes.io/projected/c3981ae9-d510-40e8-b056-9c3f963afbf4-kube-api-access-bfhhx\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.184387 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.184364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3981ae9-d510-40e8-b056-9c3f963afbf4-openshift-service-ca\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.185985 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.185962 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3981ae9-d510-40e8-b056-9c3f963afbf4-maas-api-tls\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.193977 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.193957 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhhx\" (UniqueName: \"kubernetes.io/projected/c3981ae9-d510-40e8-b056-9c3f963afbf4-kube-api-access-bfhhx\") pod \"maas-api-85cb597c75-ld6th\" (UID: \"c3981ae9-d510-40e8-b056-9c3f963afbf4\") " pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.310172 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.310100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:10.456978 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.456936 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85cb597c75-ld6th"] Jul 21 14:28:10.459633 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:28:10.459601 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3981ae9_d510_40e8_b056_9c3f963afbf4.slice/crio-e7a52dd61ce8ce06afb71d063cb7b26737c377012534c26e9577ba25787cc846 WatchSource:0}: Error finding container e7a52dd61ce8ce06afb71d063cb7b26737c377012534c26e9577ba25787cc846: Status 404 returned error can't find the container with id e7a52dd61ce8ce06afb71d063cb7b26737c377012534c26e9577ba25787cc846 Jul 21 14:28:10.539649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.539625 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-sr9z2"] Jul 21 14:28:10.544154 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.544132 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.546642 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.546622 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 14:28:10.546765 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.546629 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kbp5p\"" Jul 21 14:28:10.556711 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.556687 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-sr9z2"] Jul 21 14:28:10.687918 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.687894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5pqb\" (UniqueName: \"kubernetes.io/projected/b45e7a32-477d-4bf5-b08d-6c2c43d0e266-kube-api-access-n5pqb\") pod \"payload-processing-5bf47cd988-sr9z2\" (UID: \"b45e7a32-477d-4bf5-b08d-6c2c43d0e266\") " pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.688080 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.687939 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45e7a32-477d-4bf5-b08d-6c2c43d0e266-plugins-config-volume\") pod \"payload-processing-5bf47cd988-sr9z2\" (UID: \"b45e7a32-477d-4bf5-b08d-6c2c43d0e266\") " pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.788812 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.788781 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5pqb\" (UniqueName: \"kubernetes.io/projected/b45e7a32-477d-4bf5-b08d-6c2c43d0e266-kube-api-access-n5pqb\") pod \"payload-processing-5bf47cd988-sr9z2\" (UID: \"b45e7a32-477d-4bf5-b08d-6c2c43d0e266\") " pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.788943 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.788834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45e7a32-477d-4bf5-b08d-6c2c43d0e266-plugins-config-volume\") pod \"payload-processing-5bf47cd988-sr9z2\" (UID: \"b45e7a32-477d-4bf5-b08d-6c2c43d0e266\") " pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.789481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.789461 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45e7a32-477d-4bf5-b08d-6c2c43d0e266-plugins-config-volume\") pod \"payload-processing-5bf47cd988-sr9z2\" (UID: \"b45e7a32-477d-4bf5-b08d-6c2c43d0e266\") " pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.798909 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.798888 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5pqb\" (UniqueName: \"kubernetes.io/projected/b45e7a32-477d-4bf5-b08d-6c2c43d0e266-kube-api-access-n5pqb\") pod \"payload-processing-5bf47cd988-sr9z2\" (UID: \"b45e7a32-477d-4bf5-b08d-6c2c43d0e266\") " pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.856879 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.856852 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:10.986624 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:10.986554 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-sr9z2"] Jul 21 14:28:10.990251 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:28:10.990225 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45e7a32_477d_4bf5_b08d_6c2c43d0e266.slice/crio-a73389260d284167b1ce5287dac44ca80a8c7af9d2eb9d6c342526641f2dcde6 WatchSource:0}: Error finding container a73389260d284167b1ce5287dac44ca80a8c7af9d2eb9d6c342526641f2dcde6: Status 404 returned error can't find the container with id a73389260d284167b1ce5287dac44ca80a8c7af9d2eb9d6c342526641f2dcde6 Jul 21 14:28:11.124584 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:11.124555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" event={"ID":"c3981ae9-d510-40e8-b056-9c3f963afbf4","Type":"ContainerStarted","Data":"e7a52dd61ce8ce06afb71d063cb7b26737c377012534c26e9577ba25787cc846"} Jul 21 14:28:11.125627 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:11.125610 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" event={"ID":"b45e7a32-477d-4bf5-b08d-6c2c43d0e266","Type":"ContainerStarted","Data":"a73389260d284167b1ce5287dac44ca80a8c7af9d2eb9d6c342526641f2dcde6"} Jul 21 14:28:11.368996 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:11.368926 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:28:14.144093 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:14.144057 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" event={"ID":"b45e7a32-477d-4bf5-b08d-6c2c43d0e266","Type":"ContainerStarted","Data":"d0cdd63d788f7341adf7a98e053827fd8f68f5cb0c398dba6b050bb6455cad50"} Jul 21 14:28:14.144472 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:14.144118 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:28:14.145538 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:14.145513 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" event={"ID":"c3981ae9-d510-40e8-b056-9c3f963afbf4","Type":"ContainerStarted","Data":"ceb3646106d6f14486239f50e239689e0026e66a4984a91244d4fb6aa691a3ff"} Jul 21 14:28:14.145615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:14.145595 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:14.171818 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:14.171773 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" podStartSLOduration=1.5308354849999999 podStartE2EDuration="4.171760354s" totalImagesPullingTime="2.640924869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:28:14.16895699 +0000 UTC m=+656.536467018" watchObservedRunningTime="2026-07-21 14:28:14.171760354 +0000 UTC m=+656.539270381" Jul 21 14:28:14.190804 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:14.190764 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" podStartSLOduration=2.01834915 podStartE2EDuration="5.190751405s" totalImagesPullingTime="3.172402255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:28:14.188671433 +0000 UTC m=+656.556181460" watchObservedRunningTime="2026-07-21 14:28:14.190751405 +0000 UTC m=+656.558261433" Jul 21 14:28:20.155226 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:20.155198 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85cb597c75-ld6th" Jul 21 14:28:21.185691 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:21.185650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:28:21.282819 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:21.282784 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:28:25.152644 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:28:25.152612 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-sr9z2" Jul 21 14:29:00.611770 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:00.611723 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:29:04.756944 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.756909 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm"] Jul 21 14:29:04.762837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.762810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.767073 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.767045 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5rcx2\"" Jul 21 14:29:04.767198 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.767045 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 14:29:04.767198 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.767048 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 14:29:04.767198 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.767062 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 14:29:04.772800 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.772727 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm"] Jul 21 14:29:04.784831 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.784803 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.784955 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.784873 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.784955 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.784915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.784955 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.784940 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.785147 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.784972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7zf\" (UniqueName: \"kubernetes.io/projected/9e478ed3-a528-428e-bb26-41aa712be148-kube-api-access-gz7zf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.785147 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.785049 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e478ed3-a528-428e-bb26-41aa712be148-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.813322 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.813299 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:29:04.886360 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886338 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886467 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886467 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886388 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886467 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886406 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7zf\" (UniqueName: \"kubernetes.io/projected/9e478ed3-a528-428e-bb26-41aa712be148-kube-api-access-gz7zf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886467 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886428 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e478ed3-a528-428e-bb26-41aa712be148-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886786 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.886926 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.886885 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.888822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.888801 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e478ed3-a528-428e-bb26-41aa712be148-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.888909 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.888874 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e478ed3-a528-428e-bb26-41aa712be148-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:04.895029 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:04.894994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7zf\" (UniqueName: \"kubernetes.io/projected/9e478ed3-a528-428e-bb26-41aa712be148-kube-api-access-gz7zf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-spdxm\" (UID: \"9e478ed3-a528-428e-bb26-41aa712be148\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:05.075758 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:05.075699 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:05.206190 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:05.206163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm"] Jul 21 14:29:05.208172 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:29:05.208135 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e478ed3_a528_428e_bb26_41aa712be148.slice/crio-dddd35dcbc5674e40dc5a01624adc9ce4c33d7523fa42cb31f3a7fa2b6ade87a WatchSource:0}: Error finding container dddd35dcbc5674e40dc5a01624adc9ce4c33d7523fa42cb31f3a7fa2b6ade87a: Status 404 returned error can't find the container with id dddd35dcbc5674e40dc5a01624adc9ce4c33d7523fa42cb31f3a7fa2b6ade87a Jul 21 14:29:05.209615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:05.209595 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 14:29:05.392284 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:05.392255 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" event={"ID":"9e478ed3-a528-428e-bb26-41aa712be148","Type":"ContainerStarted","Data":"dddd35dcbc5674e40dc5a01624adc9ce4c33d7523fa42cb31f3a7fa2b6ade87a"} Jul 21 14:29:07.402797 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:07.402769 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" event={"ID":"9e478ed3-a528-428e-bb26-41aa712be148","Type":"ContainerStarted","Data":"0ec50a03c94cd22615a87a8800f94ab90072aa89e314a9ec8bc8ffbde3ee1b7d"} Jul 21 14:29:07.422732 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:07.422686 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" podStartSLOduration=2.191392055 podStartE2EDuration="3.422669838s" totalImagesPullingTime="1.231277783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:29:07.421337629 +0000 UTC m=+709.788847671" watchObservedRunningTime="2026-07-21 14:29:07.422669838 +0000 UTC m=+709.790179865" Jul 21 14:29:15.076175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:15.076062 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:15.076175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:15.076153 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:15.088390 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:15.088360 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:15.454256 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:15.454230 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-spdxm" Jul 21 14:29:20.230532 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:20.230492 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:29:27.615221 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:27.615177 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:29:40.652057 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.652019 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4"] Jul 21 14:29:40.658752 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.658728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.661474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.661446 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 14:29:40.668188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.668166 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4"] Jul 21 14:29:40.685618 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.685588 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:29:40.758846 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.758826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.758944 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.758855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.758944 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.758880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.758944 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.758933 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkhh\" (UniqueName: \"kubernetes.io/projected/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-kube-api-access-9bkhh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.759080 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.758964 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.759080 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.758991 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.859699 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.859679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.859824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.859709 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.859824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.859737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.859824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.859776 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkhh\" (UniqueName: \"kubernetes.io/projected/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-kube-api-access-9bkhh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.859824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.859815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.860055 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.859846 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.860159 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.860139 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.860218 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.860161 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.860218 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.860199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.861922 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.861901 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.862337 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.862318 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.869537 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.869514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkhh\" (UniqueName: \"kubernetes.io/projected/de6587b4-b048-4c66-bbb4-47f5d2d0b3cc-kube-api-access-9bkhh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4\" (UID: \"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:40.971892 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:40.971836 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:41.104320 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:41.104297 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4"] Jul 21 14:29:41.106097 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:29:41.106065 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6587b4_b048_4c66_bbb4_47f5d2d0b3cc.slice/crio-1d32b0a25f3690e54cb8e5d1548587cd3cb401378bbf294c826e257da3cc71de WatchSource:0}: Error finding container 1d32b0a25f3690e54cb8e5d1548587cd3cb401378bbf294c826e257da3cc71de: Status 404 returned error can't find the container with id 1d32b0a25f3690e54cb8e5d1548587cd3cb401378bbf294c826e257da3cc71de Jul 21 14:29:41.572505 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:41.572472 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" event={"ID":"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc","Type":"ContainerStarted","Data":"bdab91a282bfe9c080fb08d388461cf24a27c82b37ac89742219d242030e5bcc"} Jul 21 14:29:41.572666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:41.572521 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" event={"ID":"de6587b4-b048-4c66-bbb4-47f5d2d0b3cc","Type":"ContainerStarted","Data":"1d32b0a25f3690e54cb8e5d1548587cd3cb401378bbf294c826e257da3cc71de"} Jul 21 14:29:41.594419 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:41.594340 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" podStartSLOduration=1.362795417 podStartE2EDuration="1.594326319s" totalImagesPullingTime="231.530902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:29:41.593397496 +0000 UTC m=+743.960907554" watchObservedRunningTime="2026-07-21 14:29:41.594326319 +0000 UTC m=+743.961836346" Jul 21 14:29:50.972092 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:50.972057 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:50.972508 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:50.972134 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:50.984073 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:50.984048 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:51.635518 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:51.635490 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4" Jul 21 14:29:52.428145 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:52.428111 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:29:58.614843 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:29:58.614809 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:30:00.167623 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.167588 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g"] Jul 21 14:30:00.171388 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.171370 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" Jul 21 14:30:00.174226 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.174204 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:30:00.181027 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.180987 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" Jul 21 14:30:00.184093 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.184069 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g"] Jul 21 14:30:00.334287 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.334266 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g"] Jul 21 14:30:00.336674 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:30:00.336649 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f6f889_7dbc_4e58_bb13_07551c0abafd.slice/crio-14bc20f3e5a3bfdc9fbf92afaa7cc2947c0b3f432c2d19eff6c03bf5a2a27d82 WatchSource:0}: Error finding container 14bc20f3e5a3bfdc9fbf92afaa7cc2947c0b3f432c2d19eff6c03bf5a2a27d82: Status 404 returned error can't find the container with id 14bc20f3e5a3bfdc9fbf92afaa7cc2947c0b3f432c2d19eff6c03bf5a2a27d82 Jul 21 14:30:00.667415 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:00.667384 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" event={"ID":"f9f6f889-7dbc-4e58-bb13-07551c0abafd","Type":"ContainerStarted","Data":"14bc20f3e5a3bfdc9fbf92afaa7cc2947c0b3f432c2d19eff6c03bf5a2a27d82"} Jul 21 14:30:03.685945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:03.685910 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" event={"ID":"f9f6f889-7dbc-4e58-bb13-07551c0abafd","Type":"ContainerStarted","Data":"96c4dbd57e42096eb983a513de992d80d58ea5538a6416230970433cc1b33078"} Jul 21 14:30:03.711066 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:03.710998 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" podStartSLOduration=1.121806432 podStartE2EDuration="3.710984965s" totalImagesPullingTime="2.589178533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:30:03.710574612 +0000 UTC m=+766.078084637" watchObservedRunningTime="2026-07-21 14:30:03.710984965 +0000 UTC m=+766.078494992" Jul 21 14:30:04.691744 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:04.691710 2570 generic.go:356] "Generic (PLEG): container finished" podID="f9f6f889-7dbc-4e58-bb13-07551c0abafd" containerID="96c4dbd57e42096eb983a513de992d80d58ea5538a6416230970433cc1b33078" exitCode=0 Jul 21 14:30:04.692151 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:04.691802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" event={"ID":"f9f6f889-7dbc-4e58-bb13-07551c0abafd","Type":"ContainerDied","Data":"96c4dbd57e42096eb983a513de992d80d58ea5538a6416230970433cc1b33078"} Jul 21 14:30:05.831900 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:05.831879 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" Jul 21 14:30:06.702598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:06.702573 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" Jul 21 14:30:06.702598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:06.702581 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744070-kjs8g" event={"ID":"f9f6f889-7dbc-4e58-bb13-07551c0abafd","Type":"ContainerDied","Data":"14bc20f3e5a3bfdc9fbf92afaa7cc2947c0b3f432c2d19eff6c03bf5a2a27d82"} Jul 21 14:30:06.702786 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:30:06.702623 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14bc20f3e5a3bfdc9fbf92afaa7cc2947c0b3f432c2d19eff6c03bf5a2a27d82" Jul 21 14:31:02.443240 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:02.443200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:31:02.598896 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:02.598864 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:31:03.478618 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:03.478579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:31:03.581334 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:03.581296 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:31:04.793026 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:04.792972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:31:04.894132 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:04.894088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:31:49.145464 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.145430 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-pf4kt"] Jul 21 14:31:49.146079 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.145625 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" podUID="3f90491d-510a-45e6-98b1-db54dddffed5" containerName="manager" containerID="cri-o://89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373" gracePeriod=30 Jul 21 14:31:49.409869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.409806 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:31:49.463135 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.463107 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f90491d-510a-45e6-98b1-db54dddffed5-webhook-cert\") pod \"3f90491d-510a-45e6-98b1-db54dddffed5\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " Jul 21 14:31:49.463357 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.463335 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn8sf\" (UniqueName: \"kubernetes.io/projected/3f90491d-510a-45e6-98b1-db54dddffed5-kube-api-access-zn8sf\") pod \"3f90491d-510a-45e6-98b1-db54dddffed5\" (UID: \"3f90491d-510a-45e6-98b1-db54dddffed5\") " Jul 21 14:31:49.465734 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.465702 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f90491d-510a-45e6-98b1-db54dddffed5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3f90491d-510a-45e6-98b1-db54dddffed5" (UID: "3f90491d-510a-45e6-98b1-db54dddffed5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:31:49.466377 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.466348 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f90491d-510a-45e6-98b1-db54dddffed5-kube-api-access-zn8sf" (OuterVolumeSpecName: "kube-api-access-zn8sf") pod "3f90491d-510a-45e6-98b1-db54dddffed5" (UID: "3f90491d-510a-45e6-98b1-db54dddffed5"). InnerVolumeSpecName "kube-api-access-zn8sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:31:49.563978 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.563949 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f90491d-510a-45e6-98b1-db54dddffed5-webhook-cert\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:31:49.563978 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:49.563973 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8sf\" (UniqueName: \"kubernetes.io/projected/3f90491d-510a-45e6-98b1-db54dddffed5-kube-api-access-zn8sf\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:31:50.205152 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.205124 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f90491d-510a-45e6-98b1-db54dddffed5" containerID="89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373" exitCode=0 Jul 21 14:31:50.205666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.205162 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" Jul 21 14:31:50.205666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.205211 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" event={"ID":"3f90491d-510a-45e6-98b1-db54dddffed5","Type":"ContainerDied","Data":"89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373"} Jul 21 14:31:50.205666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.205262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-pf4kt" event={"ID":"3f90491d-510a-45e6-98b1-db54dddffed5","Type":"ContainerDied","Data":"7aa9bccc44e4f3169bd65bc45562e4d3f3df3aa796728b7c749086a9c709d1b8"} Jul 21 14:31:50.205666 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.205285 2570 scope.go:117] "RemoveContainer" containerID="89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373" Jul 21 14:31:50.213690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.213671 2570 scope.go:117] "RemoveContainer" containerID="89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373" Jul 21 14:31:50.213931 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:31:50.213914 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373\": container with ID starting with 89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373 not found: ID does not exist" containerID="89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373" Jul 21 14:31:50.213977 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.213938 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373"} err="failed to get container status \"89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373\": rpc error: code = NotFound desc = could not find container \"89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373\": container with ID starting with 89712d926c794caa287964a396ca4c626fc8ef7899b1e3a1a001a5fc3f8d5373 not found: ID does not exist" Jul 21 14:31:50.227165 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.227140 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-pf4kt"] Jul 21 14:31:50.232056 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:50.232037 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-pf4kt"] Jul 21 14:31:52.140665 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:31:52.140635 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f90491d-510a-45e6-98b1-db54dddffed5" path="/var/lib/kubelet/pods/3f90491d-510a-45e6-98b1-db54dddffed5/volumes" Jul 21 14:32:18.112600 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:18.112520 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:32:18.117002 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:18.116983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:32:19.806161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:19.806128 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:32:30.554227 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:30.554185 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:32:51.062530 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.062484 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b95d7b79-hzc55"] Jul 21 14:32:51.064934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.062954 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f90491d-510a-45e6-98b1-db54dddffed5" containerName="manager" Jul 21 14:32:51.064934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.062969 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f90491d-510a-45e6-98b1-db54dddffed5" containerName="manager" Jul 21 14:32:51.064934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.062980 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f6f889-7dbc-4e58-bb13-07551c0abafd" containerName="cleanup" Jul 21 14:32:51.064934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.062986 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f6f889-7dbc-4e58-bb13-07551c0abafd" containerName="cleanup" Jul 21 14:32:51.064934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.063073 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f6f889-7dbc-4e58-bb13-07551c0abafd" containerName="cleanup" Jul 21 14:32:51.064934 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.063085 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f90491d-510a-45e6-98b1-db54dddffed5" containerName="manager" Jul 21 14:32:51.065913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.065897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.069608 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.069586 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cpwp7\"" Jul 21 14:32:51.069608 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.069585 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 14:32:51.069769 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.069609 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 14:32:51.075074 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.075052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-hzc55"] Jul 21 14:32:51.158832 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.158786 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwntt\" (UniqueName: \"kubernetes.io/projected/a847bb8d-abfa-488c-8af3-376ac0c73fb5-kube-api-access-gwntt\") pod \"maas-controller-84b95d7b79-hzc55\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.159034 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.158845 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a847bb8d-abfa-488c-8af3-376ac0c73fb5-webhook-cert\") pod \"maas-controller-84b95d7b79-hzc55\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.259783 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.259742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwntt\" (UniqueName: \"kubernetes.io/projected/a847bb8d-abfa-488c-8af3-376ac0c73fb5-kube-api-access-gwntt\") pod \"maas-controller-84b95d7b79-hzc55\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.259783 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.259787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a847bb8d-abfa-488c-8af3-376ac0c73fb5-webhook-cert\") pod \"maas-controller-84b95d7b79-hzc55\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.262216 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.262197 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a847bb8d-abfa-488c-8af3-376ac0c73fb5-webhook-cert\") pod \"maas-controller-84b95d7b79-hzc55\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.277307 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.277279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwntt\" (UniqueName: \"kubernetes.io/projected/a847bb8d-abfa-488c-8af3-376ac0c73fb5-kube-api-access-gwntt\") pod \"maas-controller-84b95d7b79-hzc55\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.376956 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.376876 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:51.519030 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:51.518985 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-hzc55"] Jul 21 14:32:51.521244 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:32:51.521214 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847bb8d_abfa_488c_8af3_376ac0c73fb5.slice/crio-cb990fa93414c693b5843af5447f8c1994076b87c9850a17bec3a8c872934812 WatchSource:0}: Error finding container cb990fa93414c693b5843af5447f8c1994076b87c9850a17bec3a8c872934812: Status 404 returned error can't find the container with id cb990fa93414c693b5843af5447f8c1994076b87c9850a17bec3a8c872934812 Jul 21 14:32:52.497015 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:52.496975 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-hzc55" event={"ID":"a847bb8d-abfa-488c-8af3-376ac0c73fb5","Type":"ContainerStarted","Data":"cb990fa93414c693b5843af5447f8c1994076b87c9850a17bec3a8c872934812"} Jul 21 14:32:53.503628 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:53.503591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-hzc55" event={"ID":"a847bb8d-abfa-488c-8af3-376ac0c73fb5","Type":"ContainerStarted","Data":"02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b"} Jul 21 14:32:53.504081 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:53.503676 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:32:53.530995 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:32:53.530952 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b95d7b79-hzc55" podStartSLOduration=1.089023801 podStartE2EDuration="2.530938504s" totalImagesPullingTime="1.441914703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:32:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:32:53.529018076 +0000 UTC m=+935.896528095" watchObservedRunningTime="2026-07-21 14:32:53.530938504 +0000 UTC m=+935.898448555" Jul 21 14:33:04.510598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:04.510571 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:33:09.072551 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.072515 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:09.188990 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.188955 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-hzc55"] Jul 21 14:33:09.189164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.189143 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84b95d7b79-hzc55" podUID="a847bb8d-abfa-488c-8af3-376ac0c73fb5" containerName="manager" containerID="cri-o://02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b" gracePeriod=30 Jul 21 14:33:09.439262 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.439238 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:33:09.517019 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.516974 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a847bb8d-abfa-488c-8af3-376ac0c73fb5-webhook-cert\") pod \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " Jul 21 14:33:09.517165 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.517044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwntt\" (UniqueName: \"kubernetes.io/projected/a847bb8d-abfa-488c-8af3-376ac0c73fb5-kube-api-access-gwntt\") pod \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\" (UID: \"a847bb8d-abfa-488c-8af3-376ac0c73fb5\") " Jul 21 14:33:09.518802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.518771 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a847bb8d-abfa-488c-8af3-376ac0c73fb5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a847bb8d-abfa-488c-8af3-376ac0c73fb5" (UID: "a847bb8d-abfa-488c-8af3-376ac0c73fb5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:33:09.518998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.518979 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a847bb8d-abfa-488c-8af3-376ac0c73fb5-kube-api-access-gwntt" (OuterVolumeSpecName: "kube-api-access-gwntt") pod "a847bb8d-abfa-488c-8af3-376ac0c73fb5" (UID: "a847bb8d-abfa-488c-8af3-376ac0c73fb5"). InnerVolumeSpecName "kube-api-access-gwntt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:33:09.579589 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.579555 2570 generic.go:356] "Generic (PLEG): container finished" podID="a847bb8d-abfa-488c-8af3-376ac0c73fb5" containerID="02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b" exitCode=0 Jul 21 14:33:09.579724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.579596 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-hzc55" Jul 21 14:33:09.579724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.579603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-hzc55" event={"ID":"a847bb8d-abfa-488c-8af3-376ac0c73fb5","Type":"ContainerDied","Data":"02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b"} Jul 21 14:33:09.579724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.579653 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-hzc55" event={"ID":"a847bb8d-abfa-488c-8af3-376ac0c73fb5","Type":"ContainerDied","Data":"cb990fa93414c693b5843af5447f8c1994076b87c9850a17bec3a8c872934812"} Jul 21 14:33:09.579724 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.579675 2570 scope.go:117] "RemoveContainer" containerID="02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b" Jul 21 14:33:09.589364 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.589345 2570 scope.go:117] "RemoveContainer" containerID="02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b" Jul 21 14:33:09.589618 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:33:09.589595 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b\": container with ID starting with 02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b not found: ID does not exist" containerID="02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b" Jul 21 14:33:09.589698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.589624 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b"} err="failed to get container status \"02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b\": rpc error: code = NotFound desc = could not find container \"02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b\": container with ID starting with 02453139c8efbc0980cb2f929907bfb6f95c6dd48e91a985eb1d936e81af0f5b not found: ID does not exist" Jul 21 14:33:09.604049 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.604026 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-hzc55"] Jul 21 14:33:09.608514 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.608464 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-hzc55"] Jul 21 14:33:09.618127 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.618100 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a847bb8d-abfa-488c-8af3-376ac0c73fb5-webhook-cert\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:33:09.618127 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:09.618126 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwntt\" (UniqueName: \"kubernetes.io/projected/a847bb8d-abfa-488c-8af3-376ac0c73fb5-kube-api-access-gwntt\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:33:10.141548 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.141514 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a847bb8d-abfa-488c-8af3-376ac0c73fb5" path="/var/lib/kubelet/pods/a847bb8d-abfa-488c-8af3-376ac0c73fb5/volumes" Jul 21 14:33:10.301466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.301420 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:10.420115 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.420033 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84b95d7b79-45qnw"] Jul 21 14:33:10.420473 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.420458 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a847bb8d-abfa-488c-8af3-376ac0c73fb5" containerName="manager" Jul 21 14:33:10.420509 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.420475 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a847bb8d-abfa-488c-8af3-376ac0c73fb5" containerName="manager" Jul 21 14:33:10.420570 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.420560 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a847bb8d-abfa-488c-8af3-376ac0c73fb5" containerName="manager" Jul 21 14:33:10.424870 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.424845 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.427591 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.427573 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 14:33:10.427811 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.427790 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cpwp7\"" Jul 21 14:33:10.427898 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.427831 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 14:33:10.433759 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.433728 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-45qnw"] Jul 21 14:33:10.526380 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.526348 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw926\" (UniqueName: \"kubernetes.io/projected/76522a94-1e17-4d4c-9107-003dc76df617-kube-api-access-lw926\") pod \"maas-controller-84b95d7b79-45qnw\" (UID: \"76522a94-1e17-4d4c-9107-003dc76df617\") " pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.526534 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.526398 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76522a94-1e17-4d4c-9107-003dc76df617-webhook-cert\") pod \"maas-controller-84b95d7b79-45qnw\" (UID: \"76522a94-1e17-4d4c-9107-003dc76df617\") " pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.627499 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.627468 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76522a94-1e17-4d4c-9107-003dc76df617-webhook-cert\") pod \"maas-controller-84b95d7b79-45qnw\" (UID: \"76522a94-1e17-4d4c-9107-003dc76df617\") " pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.627690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.627548 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw926\" (UniqueName: \"kubernetes.io/projected/76522a94-1e17-4d4c-9107-003dc76df617-kube-api-access-lw926\") pod \"maas-controller-84b95d7b79-45qnw\" (UID: \"76522a94-1e17-4d4c-9107-003dc76df617\") " pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.629889 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.629863 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76522a94-1e17-4d4c-9107-003dc76df617-webhook-cert\") pod \"maas-controller-84b95d7b79-45qnw\" (UID: \"76522a94-1e17-4d4c-9107-003dc76df617\") " pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.637876 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.637853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw926\" (UniqueName: \"kubernetes.io/projected/76522a94-1e17-4d4c-9107-003dc76df617-kube-api-access-lw926\") pod \"maas-controller-84b95d7b79-45qnw\" (UID: \"76522a94-1e17-4d4c-9107-003dc76df617\") " pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.735692 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.735609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:10.879371 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:10.879342 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84b95d7b79-45qnw"] Jul 21 14:33:10.881221 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:33:10.881194 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76522a94_1e17_4d4c_9107_003dc76df617.slice/crio-7c06d748663c725eddbefc6d4b6212e7b7a3cb68c824225a48800408654eea4a WatchSource:0}: Error finding container 7c06d748663c725eddbefc6d4b6212e7b7a3cb68c824225a48800408654eea4a: Status 404 returned error can't find the container with id 7c06d748663c725eddbefc6d4b6212e7b7a3cb68c824225a48800408654eea4a Jul 21 14:33:11.591516 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:11.591481 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-45qnw" event={"ID":"76522a94-1e17-4d4c-9107-003dc76df617","Type":"ContainerStarted","Data":"f230926df5f8455f3977e2f4338afe65c056e26d13dd0b1322cdb35667a9d2f5"} Jul 21 14:33:11.591905 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:11.591563 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84b95d7b79-45qnw" event={"ID":"76522a94-1e17-4d4c-9107-003dc76df617","Type":"ContainerStarted","Data":"7c06d748663c725eddbefc6d4b6212e7b7a3cb68c824225a48800408654eea4a"} Jul 21 14:33:11.591905 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:11.591601 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:11.612894 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:11.612824 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84b95d7b79-45qnw" podStartSLOduration=1.044730953 podStartE2EDuration="1.612809596s" totalImagesPullingTime="568.078643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:33:11.611954658 +0000 UTC m=+953.979464688" watchObservedRunningTime="2026-07-21 14:33:11.612809596 +0000 UTC m=+953.980319623" Jul 21 14:33:22.599375 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:22.599347 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84b95d7b79-45qnw" Jul 21 14:33:44.186252 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:44.186150 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:45.692540 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:45.692510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:51.183856 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:51.183818 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:51.328151 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:51.328109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:51.470241 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:51.470156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:51.600785 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:51.600740 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:58.584845 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:58.584810 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:33:58.738475 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:33:58.738442 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:34:34.661564 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:34:34.661531 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:36:09.205838 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:36:09.205797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:37:18.142762 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:18.142737 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:37:18.148061 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:18.148041 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:37:51.418548 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:51.418518 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf"] Jul 21 14:37:51.420897 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:51.418709 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" podUID="542fd1e5-2c0f-4377-a994-75f049860445" containerName="manager" containerID="cri-o://6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa" gracePeriod=10 Jul 21 14:37:54.081973 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.081946 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:37:54.222906 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.222872 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg6kz\" (UniqueName: \"kubernetes.io/projected/542fd1e5-2c0f-4377-a994-75f049860445-kube-api-access-hg6kz\") pod \"542fd1e5-2c0f-4377-a994-75f049860445\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " Jul 21 14:37:54.223102 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.222962 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/542fd1e5-2c0f-4377-a994-75f049860445-extensions-socket-volume\") pod \"542fd1e5-2c0f-4377-a994-75f049860445\" (UID: \"542fd1e5-2c0f-4377-a994-75f049860445\") " Jul 21 14:37:54.223356 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.223332 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542fd1e5-2c0f-4377-a994-75f049860445-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "542fd1e5-2c0f-4377-a994-75f049860445" (UID: "542fd1e5-2c0f-4377-a994-75f049860445"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:37:54.224970 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.224949 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542fd1e5-2c0f-4377-a994-75f049860445-kube-api-access-hg6kz" (OuterVolumeSpecName: "kube-api-access-hg6kz") pod "542fd1e5-2c0f-4377-a994-75f049860445" (UID: "542fd1e5-2c0f-4377-a994-75f049860445"). InnerVolumeSpecName "kube-api-access-hg6kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:37:54.323796 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.323767 2570 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/542fd1e5-2c0f-4377-a994-75f049860445-extensions-socket-volume\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:37:54.323796 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.323791 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6kz\" (UniqueName: \"kubernetes.io/projected/542fd1e5-2c0f-4377-a994-75f049860445-kube-api-access-hg6kz\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:37:54.936136 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.936102 2570 generic.go:356] "Generic (PLEG): container finished" podID="542fd1e5-2c0f-4377-a994-75f049860445" containerID="6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa" exitCode=0 Jul 21 14:37:54.936341 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.936142 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" Jul 21 14:37:54.936341 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.936186 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" event={"ID":"542fd1e5-2c0f-4377-a994-75f049860445","Type":"ContainerDied","Data":"6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa"} Jul 21 14:37:54.936341 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.936241 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf" event={"ID":"542fd1e5-2c0f-4377-a994-75f049860445","Type":"ContainerDied","Data":"cdcae0e9c6c90b509fb2ee4d0c6e86ccd44b4dff6e90ea90067a51191b0dab5e"} Jul 21 14:37:54.936341 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.936263 2570 scope.go:117] "RemoveContainer" containerID="6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa" Jul 21 14:37:54.945884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.945865 2570 scope.go:117] "RemoveContainer" containerID="6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa" Jul 21 14:37:54.946165 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:37:54.946129 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa\": container with ID starting with 6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa not found: ID does not exist" containerID="6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa" Jul 21 14:37:54.946239 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.946174 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa"} err="failed to get container status \"6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa\": rpc error: code = NotFound desc = could not find container \"6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa\": container with ID starting with 6ae6cdff19ce735cd7fa02bddc4ab4512cb44b8550b79c951b3ecf487dd839fa not found: ID does not exist" Jul 21 14:37:54.960396 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.960374 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf"] Jul 21 14:37:54.965236 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:54.965214 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6vwzf"] Jul 21 14:37:56.141506 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:37:56.141474 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542fd1e5-2c0f-4377-a994-75f049860445" path="/var/lib/kubelet/pods/542fd1e5-2c0f-4377-a994-75f049860445/volumes" Jul 21 14:38:48.274768 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:48.274729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:38:52.686697 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.686654 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52"] Jul 21 14:38:52.687247 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.687225 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542fd1e5-2c0f-4377-a994-75f049860445" containerName="manager" Jul 21 14:38:52.687326 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.687250 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="542fd1e5-2c0f-4377-a994-75f049860445" containerName="manager" Jul 21 14:38:52.687403 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.687389 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="542fd1e5-2c0f-4377-a994-75f049860445" containerName="manager" Jul 21 14:38:52.690726 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.690700 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:52.693157 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.693134 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-82jnm\"" Jul 21 14:38:52.707018 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.706984 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52"] Jul 21 14:38:52.781176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.781149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927c6\" (UniqueName: \"kubernetes.io/projected/32aa0b2a-a811-47eb-8f26-6e5a77bb0b17-kube-api-access-927c6\") pod \"kuadrant-operator-controller-manager-86878c769f-xls52\" (UID: \"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:52.781290 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.781187 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32aa0b2a-a811-47eb-8f26-6e5a77bb0b17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xls52\" (UID: \"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:52.882134 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.882101 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927c6\" (UniqueName: \"kubernetes.io/projected/32aa0b2a-a811-47eb-8f26-6e5a77bb0b17-kube-api-access-927c6\") pod \"kuadrant-operator-controller-manager-86878c769f-xls52\" (UID: \"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:52.882254 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.882146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32aa0b2a-a811-47eb-8f26-6e5a77bb0b17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xls52\" (UID: \"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:52.882517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.882496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32aa0b2a-a811-47eb-8f26-6e5a77bb0b17-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-xls52\" (UID: \"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:52.892108 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:52.892090 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-927c6\" (UniqueName: \"kubernetes.io/projected/32aa0b2a-a811-47eb-8f26-6e5a77bb0b17-kube-api-access-927c6\") pod \"kuadrant-operator-controller-manager-86878c769f-xls52\" (UID: \"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:53.001738 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:53.001683 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:53.140469 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:53.140415 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52"] Jul 21 14:38:53.143148 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:38:53.143121 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32aa0b2a_a811_47eb_8f26_6e5a77bb0b17.slice/crio-f0ad83fcdbd92d1bc246b5d628a8a8da3e97e66787d6024899cd86e833c97503 WatchSource:0}: Error finding container f0ad83fcdbd92d1bc246b5d628a8a8da3e97e66787d6024899cd86e833c97503: Status 404 returned error can't find the container with id f0ad83fcdbd92d1bc246b5d628a8a8da3e97e66787d6024899cd86e833c97503 Jul 21 14:38:53.225630 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:53.225599 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" event={"ID":"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17","Type":"ContainerStarted","Data":"2bc79b5c4f4a8830a95b065e6c7bb637ec33f6d22ad49c9fae064aa16acf5211"} Jul 21 14:38:53.225749 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:53.225664 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:38:53.225749 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:53.225690 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" event={"ID":"32aa0b2a-a811-47eb-8f26-6e5a77bb0b17","Type":"ContainerStarted","Data":"f0ad83fcdbd92d1bc246b5d628a8a8da3e97e66787d6024899cd86e833c97503"} Jul 21 14:38:53.248384 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:38:53.248347 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" podStartSLOduration=1.248332377 podStartE2EDuration="1.248332377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:38:53.247325275 +0000 UTC m=+1295.614835304" watchObservedRunningTime="2026-07-21 14:38:53.248332377 +0000 UTC m=+1295.615842404" Jul 21 14:39:04.231959 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:39:04.231930 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-xls52" Jul 21 14:41:07.648825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:41:07.648787 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:07.734160 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:41:07.734132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:44.816468 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:41:44.816432 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:41:57.692298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:41:57.692252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:12.403113 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:42:12.403078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:18.180819 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:42:18.180789 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:42:18.183967 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:42:18.183947 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:42:43.841561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:42:43.841486 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:42:44.594288 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:42:44.594257 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:13.973433 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:43:13.973395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:38.059164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:43:38.059131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:43:44.108366 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:43:44.108332 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:44:13.628356 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:44:13.628281 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:00.141431 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:00.141403 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp"] Jul 21 14:45:00.144907 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:00.144889 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" Jul 21 14:45:00.147304 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:00.147282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:45:00.154593 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:00.154575 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" Jul 21 14:45:00.164435 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:00.164413 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp"] Jul 21 14:45:00.494981 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:00.494953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp"] Jul 21 14:45:00.497346 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:45:00.497311 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c9b1d4_6843_4ab4_a3a8_21d6323b9596.slice/crio-2bd20b8c58ae6ddb027af0542bb49c6fced813b5f2c902517110a692a752f791 WatchSource:0}: Error finding container 2bd20b8c58ae6ddb027af0542bb49c6fced813b5f2c902517110a692a752f791: Status 404 returned error can't find the container with id 2bd20b8c58ae6ddb027af0542bb49c6fced813b5f2c902517110a692a752f791 Jul 21 14:45:01.018791 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:01.018757 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" event={"ID":"29c9b1d4-6843-4ab4-a3a8-21d6323b9596","Type":"ContainerStarted","Data":"60060638feacd6910ab9193156f878d826348044b5e643bf90ef0441a47a4b6d"} Jul 21 14:45:01.018956 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:01.018812 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" event={"ID":"29c9b1d4-6843-4ab4-a3a8-21d6323b9596","Type":"ContainerStarted","Data":"2bd20b8c58ae6ddb027af0542bb49c6fced813b5f2c902517110a692a752f791"} Jul 21 14:45:01.038199 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:01.038155 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" podStartSLOduration=1.03814125 podStartE2EDuration="1.03814125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:45:01.03613495 +0000 UTC m=+1663.403644977" watchObservedRunningTime="2026-07-21 14:45:01.03814125 +0000 UTC m=+1663.405651279" Jul 21 14:45:02.024819 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:02.024788 2570 generic.go:356] "Generic (PLEG): container finished" podID="29c9b1d4-6843-4ab4-a3a8-21d6323b9596" containerID="60060638feacd6910ab9193156f878d826348044b5e643bf90ef0441a47a4b6d" exitCode=0 Jul 21 14:45:02.025239 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:02.024874 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" event={"ID":"29c9b1d4-6843-4ab4-a3a8-21d6323b9596","Type":"ContainerDied","Data":"60060638feacd6910ab9193156f878d826348044b5e643bf90ef0441a47a4b6d"} Jul 21 14:45:03.166631 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:03.166610 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" Jul 21 14:45:04.036563 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:04.036529 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" Jul 21 14:45:04.036735 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:04.036561 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744085-wxszp" event={"ID":"29c9b1d4-6843-4ab4-a3a8-21d6323b9596","Type":"ContainerDied","Data":"2bd20b8c58ae6ddb027af0542bb49c6fced813b5f2c902517110a692a752f791"} Jul 21 14:45:04.036735 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:04.036608 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd20b8c58ae6ddb027af0542bb49c6fced813b5f2c902517110a692a752f791" Jul 21 14:45:51.011615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:51.011540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:51.125159 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:51.125115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:51.418837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:51.418804 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:58.923146 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:58.923115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:59.017400 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:59.017375 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:59.402588 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:59.402552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:59.493636 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:59.493605 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:45:59.694422 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:45:59.694356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:01.819588 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:01.819552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:01.965873 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:01.965844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:02.061983 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:02.061943 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:02.214150 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:02.214121 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:02.655202 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:02.655171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:02.753274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:02.753221 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:27.383558 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.383527 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7"] Jul 21 14:46:27.383995 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.383920 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c9b1d4-6843-4ab4-a3a8-21d6323b9596" containerName="cleanup" Jul 21 14:46:27.383995 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.383932 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c9b1d4-6843-4ab4-a3a8-21d6323b9596" containerName="cleanup" Jul 21 14:46:27.384112 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.384025 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c9b1d4-6843-4ab4-a3a8-21d6323b9596" containerName="cleanup" Jul 21 14:46:27.386957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.386939 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:27.390333 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.390312 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:46:27.397500 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.397478 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7"] Jul 21 14:46:27.459903 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.459880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f02753-004c-4171-b4db-9fcc57ab6922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-78a7b505-klqk7\" (UID: \"07f02753-004c-4171-b4db-9fcc57ab6922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:27.560828 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.560802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f02753-004c-4171-b4db-9fcc57ab6922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-78a7b505-klqk7\" (UID: \"07f02753-004c-4171-b4db-9fcc57ab6922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:27.561373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.561356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f02753-004c-4171-b4db-9fcc57ab6922-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-78a7b505-klqk7\" (UID: \"07f02753-004c-4171-b4db-9fcc57ab6922\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:27.697067 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.697035 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:27.833919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:27.833848 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7"] Jul 21 14:46:27.836248 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:46:27.836220 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f02753_004c_4171_b4db_9fcc57ab6922.slice/crio-ee1bb4f6c090b0da26c1080183aefb847744856e09c0f896704ea39baf302492 WatchSource:0}: Error finding container ee1bb4f6c090b0da26c1080183aefb847744856e09c0f896704ea39baf302492: Status 404 returned error can't find the container with id ee1bb4f6c090b0da26c1080183aefb847744856e09c0f896704ea39baf302492 Jul 21 14:46:28.435940 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:28.435865 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" event={"ID":"07f02753-004c-4171-b4db-9fcc57ab6922","Type":"ContainerStarted","Data":"e8b11f089c81963c284be4331fa1a145c456b16b15ebe327202bdc60fed899da"} Jul 21 14:46:28.435940 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:28.435923 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" event={"ID":"07f02753-004c-4171-b4db-9fcc57ab6922","Type":"ContainerStarted","Data":"ee1bb4f6c090b0da26c1080183aefb847744856e09c0f896704ea39baf302492"} Jul 21 14:46:28.457764 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:28.457720 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" podStartSLOduration=1.457706141 podStartE2EDuration="1.457706141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:46:28.45478363 +0000 UTC m=+1750.822293656" watchObservedRunningTime="2026-07-21 14:46:28.457706141 +0000 UTC m=+1750.825216196" Jul 21 14:46:29.441758 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:29.441724 2570 generic.go:356] "Generic (PLEG): container finished" podID="07f02753-004c-4171-b4db-9fcc57ab6922" containerID="e8b11f089c81963c284be4331fa1a145c456b16b15ebe327202bdc60fed899da" exitCode=0 Jul 21 14:46:29.442132 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:29.441790 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" event={"ID":"07f02753-004c-4171-b4db-9fcc57ab6922","Type":"ContainerDied","Data":"e8b11f089c81963c284be4331fa1a145c456b16b15ebe327202bdc60fed899da"} Jul 21 14:46:30.586563 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:30.586534 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:30.683162 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:30.683140 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f02753-004c-4171-b4db-9fcc57ab6922-maas-api-service-ca\") pod \"07f02753-004c-4171-b4db-9fcc57ab6922\" (UID: \"07f02753-004c-4171-b4db-9fcc57ab6922\") " Jul 21 14:46:30.683444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:30.683425 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f02753-004c-4171-b4db-9fcc57ab6922-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07f02753-004c-4171-b4db-9fcc57ab6922" (UID: "07f02753-004c-4171-b4db-9fcc57ab6922"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:46:30.784204 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:30.784124 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07f02753-004c-4171-b4db-9fcc57ab6922-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:46:31.454055 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:31.454026 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" Jul 21 14:46:31.454234 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:31.454056 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7" event={"ID":"07f02753-004c-4171-b4db-9fcc57ab6922","Type":"ContainerDied","Data":"ee1bb4f6c090b0da26c1080183aefb847744856e09c0f896704ea39baf302492"} Jul 21 14:46:31.454234 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:31.454107 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1bb4f6c090b0da26c1080183aefb847744856e09c0f896704ea39baf302492" Jul 21 14:46:38.175838 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:38.175784 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:42.509188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:42.509158 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7"] Jul 21 14:46:42.515219 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:42.515193 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-78a7b505-klqk7"] Jul 21 14:46:42.529846 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:42.529820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:42.692702 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:42.692676 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:42.913189 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:42.913163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:43.172029 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:43.171941 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:43.273451 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:43.273416 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:43.400893 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:43.400865 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:43.755567 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:43.755538 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:44.091463 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:44.091395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:44.141855 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:44.141825 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07f02753-004c-4171-b4db-9fcc57ab6922" path="/var/lib/kubelet/pods/07f02753-004c-4171-b4db-9fcc57ab6922/volumes" Jul 21 14:46:46.245074 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:46.245043 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:46.467132 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:46.467090 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:46.562889 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:46.562814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:47.114110 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:47.114078 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:46:47.240368 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:46:47.240341 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:11.331738 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.331698 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2"] Jul 21 14:47:11.332201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.332140 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07f02753-004c-4171-b4db-9fcc57ab6922" containerName="revoke-keys" Jul 21 14:47:11.332201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.332152 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="07f02753-004c-4171-b4db-9fcc57ab6922" containerName="revoke-keys" Jul 21 14:47:11.332278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.332212 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="07f02753-004c-4171-b4db-9fcc57ab6922" containerName="revoke-keys" Jul 21 14:47:11.335225 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.335202 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:11.338277 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.338254 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:47:11.346958 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.346932 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2"] Jul 21 14:47:11.427883 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.427840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a13053b-c81b-4cd3-9edc-d346c193ad88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2\" (UID: \"4a13053b-c81b-4cd3-9edc-d346c193ad88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:11.529075 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.528998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a13053b-c81b-4cd3-9edc-d346c193ad88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2\" (UID: \"4a13053b-c81b-4cd3-9edc-d346c193ad88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:11.529646 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.529624 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a13053b-c81b-4cd3-9edc-d346c193ad88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2\" (UID: \"4a13053b-c81b-4cd3-9edc-d346c193ad88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:11.646134 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.646100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:11.996840 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:11.996816 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2"] Jul 21 14:47:11.998772 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:47:11.998740 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a13053b_c81b_4cd3_9edc_d346c193ad88.slice/crio-baf7eb81e6c92162bab1d4d5ec7d7a5b2916f01d9222f826a0a243a818d62aa2 WatchSource:0}: Error finding container baf7eb81e6c92162bab1d4d5ec7d7a5b2916f01d9222f826a0a243a818d62aa2: Status 404 returned error can't find the container with id baf7eb81e6c92162bab1d4d5ec7d7a5b2916f01d9222f826a0a243a818d62aa2 Jul 21 14:47:12.651641 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:12.651604 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" event={"ID":"4a13053b-c81b-4cd3-9edc-d346c193ad88","Type":"ContainerStarted","Data":"9e43bdc2afcdc1d2c7374816dd3e5a3d98418b64b945dcdc2ccf1bdd31ac37bc"} Jul 21 14:47:12.652034 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:12.651668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" event={"ID":"4a13053b-c81b-4cd3-9edc-d346c193ad88","Type":"ContainerStarted","Data":"baf7eb81e6c92162bab1d4d5ec7d7a5b2916f01d9222f826a0a243a818d62aa2"} Jul 21 14:47:12.673149 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:12.673104 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" podStartSLOduration=1.673089814 podStartE2EDuration="1.673089814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:47:12.671159794 +0000 UTC m=+1795.038669822" watchObservedRunningTime="2026-07-21 14:47:12.673089814 +0000 UTC m=+1795.040599842" Jul 21 14:47:13.659677 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:13.659644 2570 generic.go:356] "Generic (PLEG): container finished" podID="4a13053b-c81b-4cd3-9edc-d346c193ad88" containerID="9e43bdc2afcdc1d2c7374816dd3e5a3d98418b64b945dcdc2ccf1bdd31ac37bc" exitCode=0 Jul 21 14:47:13.660062 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:13.659698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" event={"ID":"4a13053b-c81b-4cd3-9edc-d346c193ad88","Type":"ContainerDied","Data":"9e43bdc2afcdc1d2c7374816dd3e5a3d98418b64b945dcdc2ccf1bdd31ac37bc"} Jul 21 14:47:14.805831 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:14.805806 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:14.959064 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:14.958946 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a13053b-c81b-4cd3-9edc-d346c193ad88-maas-api-service-ca\") pod \"4a13053b-c81b-4cd3-9edc-d346c193ad88\" (UID: \"4a13053b-c81b-4cd3-9edc-d346c193ad88\") " Jul 21 14:47:14.959329 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:14.959305 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a13053b-c81b-4cd3-9edc-d346c193ad88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a13053b-c81b-4cd3-9edc-d346c193ad88" (UID: "4a13053b-c81b-4cd3-9edc-d346c193ad88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:47:15.060370 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:15.060337 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a13053b-c81b-4cd3-9edc-d346c193ad88-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:47:15.671536 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:15.671496 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" Jul 21 14:47:15.671775 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:15.671538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2" event={"ID":"4a13053b-c81b-4cd3-9edc-d346c193ad88","Type":"ContainerDied","Data":"baf7eb81e6c92162bab1d4d5ec7d7a5b2916f01d9222f826a0a243a818d62aa2"} Jul 21 14:47:15.671775 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:15.671597 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baf7eb81e6c92162bab1d4d5ec7d7a5b2916f01d9222f826a0a243a818d62aa2" Jul 21 14:47:18.212078 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:18.212048 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:47:18.215776 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:18.215755 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:47:22.121627 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:22.121597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:26.398211 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:26.398181 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2"] Jul 21 14:47:26.401518 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:26.401497 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-98cb868b-snwj2"] Jul 21 14:47:26.412543 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:26.412500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:26.564129 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:26.564102 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:26.798491 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:26.798429 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:27.055284 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:27.055218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:27.424840 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:27.424805 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:27.678220 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:27.678137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:28.141214 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:28.141177 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a13053b-c81b-4cd3-9edc-d346c193ad88" path="/var/lib/kubelet/pods/4a13053b-c81b-4cd3-9edc-d346c193ad88/volumes" Jul 21 14:47:29.365481 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:29.365446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:29.473935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:29.473898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:29.766461 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:29.766432 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:29.898287 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:29.898256 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:30.546205 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:30.546158 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:30.648487 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:30.648458 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:47:55.139669 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.139628 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs"] Jul 21 14:47:55.140307 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.140150 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a13053b-c81b-4cd3-9edc-d346c193ad88" containerName="revoke-keys" Jul 21 14:47:55.140307 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.140164 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a13053b-c81b-4cd3-9edc-d346c193ad88" containerName="revoke-keys" Jul 21 14:47:55.140307 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.140233 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a13053b-c81b-4cd3-9edc-d346c193ad88" containerName="revoke-keys" Jul 21 14:47:55.143245 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.143229 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:55.146094 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.146071 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:47:55.152795 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.152770 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs"] Jul 21 14:47:55.273157 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.273130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c72b012-bf2e-49ba-afb4-d3facdfed2ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs\" (UID: \"7c72b012-bf2e-49ba-afb4-d3facdfed2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:55.373698 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.373665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c72b012-bf2e-49ba-afb4-d3facdfed2ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs\" (UID: \"7c72b012-bf2e-49ba-afb4-d3facdfed2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:55.374375 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.374349 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c72b012-bf2e-49ba-afb4-d3facdfed2ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs\" (UID: \"7c72b012-bf2e-49ba-afb4-d3facdfed2ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:55.455596 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.455523 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:55.592284 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.592254 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs"] Jul 21 14:47:55.594769 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:47:55.594733 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c72b012_bf2e_49ba_afb4_d3facdfed2ff.slice/crio-f9a4a70a871733338ceee8ae61d7ebcc264237f59c08c5b02f3a54ef2d4aaba6 WatchSource:0}: Error finding container f9a4a70a871733338ceee8ae61d7ebcc264237f59c08c5b02f3a54ef2d4aaba6: Status 404 returned error can't find the container with id f9a4a70a871733338ceee8ae61d7ebcc264237f59c08c5b02f3a54ef2d4aaba6 Jul 21 14:47:55.870444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.870415 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" event={"ID":"7c72b012-bf2e-49ba-afb4-d3facdfed2ff","Type":"ContainerStarted","Data":"c6a6e7d91120d86bca5ce3471d633deeabd35a3e70bc0953ed584a80e0b3ab35"} Jul 21 14:47:55.870559 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.870473 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" event={"ID":"7c72b012-bf2e-49ba-afb4-d3facdfed2ff","Type":"ContainerStarted","Data":"f9a4a70a871733338ceee8ae61d7ebcc264237f59c08c5b02f3a54ef2d4aaba6"} Jul 21 14:47:55.890345 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:55.890300 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" podStartSLOduration=0.890286573 podStartE2EDuration="890.286573ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:47:55.888843078 +0000 UTC m=+1838.256353106" watchObservedRunningTime="2026-07-21 14:47:55.890286573 +0000 UTC m=+1838.257796600" Jul 21 14:47:56.877997 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:56.877955 2570 generic.go:356] "Generic (PLEG): container finished" podID="7c72b012-bf2e-49ba-afb4-d3facdfed2ff" containerID="c6a6e7d91120d86bca5ce3471d633deeabd35a3e70bc0953ed584a80e0b3ab35" exitCode=0 Jul 21 14:47:56.878372 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:56.878052 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" event={"ID":"7c72b012-bf2e-49ba-afb4-d3facdfed2ff","Type":"ContainerDied","Data":"c6a6e7d91120d86bca5ce3471d633deeabd35a3e70bc0953ed584a80e0b3ab35"} Jul 21 14:47:58.014732 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.014708 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:58.094078 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.094053 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c72b012-bf2e-49ba-afb4-d3facdfed2ff-maas-api-service-ca\") pod \"7c72b012-bf2e-49ba-afb4-d3facdfed2ff\" (UID: \"7c72b012-bf2e-49ba-afb4-d3facdfed2ff\") " Jul 21 14:47:58.094389 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.094365 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c72b012-bf2e-49ba-afb4-d3facdfed2ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c72b012-bf2e-49ba-afb4-d3facdfed2ff" (UID: "7c72b012-bf2e-49ba-afb4-d3facdfed2ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:47:58.194794 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.194768 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c72b012-bf2e-49ba-afb4-d3facdfed2ff-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:47:58.888853 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.888816 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" Jul 21 14:47:58.889080 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.888847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs" event={"ID":"7c72b012-bf2e-49ba-afb4-d3facdfed2ff","Type":"ContainerDied","Data":"f9a4a70a871733338ceee8ae61d7ebcc264237f59c08c5b02f3a54ef2d4aaba6"} Jul 21 14:47:58.889080 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:47:58.888920 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9a4a70a871733338ceee8ae61d7ebcc264237f59c08c5b02f3a54ef2d4aaba6" Jul 21 14:48:05.920358 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:05.920323 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:10.253861 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:10.253824 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs"] Jul 21 14:48:10.256555 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:10.256532 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a74b282-x4xzs"] Jul 21 14:48:10.258942 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:10.258914 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:11.463598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:11.463561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:11.675401 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:11.675372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:11.884872 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:11.884800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:12.143417 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:12.143384 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c72b012-bf2e-49ba-afb4-d3facdfed2ff" path="/var/lib/kubelet/pods/7c72b012-bf2e-49ba-afb4-d3facdfed2ff/volumes" Jul 21 14:48:12.205761 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:12.205730 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:12.361288 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:12.361254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:14.666649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:14.666615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:14.764234 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:14.764203 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:14.858882 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:14.858851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:14.968139 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:14.968072 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:15.544409 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:15.544377 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:15.672663 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:15.672634 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:31.922676 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:31.922642 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:32.028509 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:32.028474 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:36.639170 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:36.639138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:36.739682 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:36.739656 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:39.461545 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.461510 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj"] Jul 21 14:48:39.461905 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.461896 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c72b012-bf2e-49ba-afb4-d3facdfed2ff" containerName="revoke-keys" Jul 21 14:48:39.461946 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.461907 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c72b012-bf2e-49ba-afb4-d3facdfed2ff" containerName="revoke-keys" Jul 21 14:48:39.461980 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.461973 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c72b012-bf2e-49ba-afb4-d3facdfed2ff" containerName="revoke-keys" Jul 21 14:48:39.465134 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.465113 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:39.467702 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.467681 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:48:39.474407 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.474386 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj"] Jul 21 14:48:39.506930 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.506900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b180ec-0fe8-4af2-b9a3-81d2216ad865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj\" (UID: \"95b180ec-0fe8-4af2-b9a3-81d2216ad865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:39.608125 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.608098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b180ec-0fe8-4af2-b9a3-81d2216ad865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj\" (UID: \"95b180ec-0fe8-4af2-b9a3-81d2216ad865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:39.608714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.608689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b180ec-0fe8-4af2-b9a3-81d2216ad865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj\" (UID: \"95b180ec-0fe8-4af2-b9a3-81d2216ad865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:39.777627 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.777564 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:39.916552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:39.916523 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj"] Jul 21 14:48:39.918242 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:48:39.918218 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b180ec_0fe8_4af2_b9a3_81d2216ad865.slice/crio-ea2366595d97abc79dba6c67e610750806634f83af58dd2f81041324e739482d WatchSource:0}: Error finding container ea2366595d97abc79dba6c67e610750806634f83af58dd2f81041324e739482d: Status 404 returned error can't find the container with id ea2366595d97abc79dba6c67e610750806634f83af58dd2f81041324e739482d Jul 21 14:48:40.098782 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:40.098714 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" event={"ID":"95b180ec-0fe8-4af2-b9a3-81d2216ad865","Type":"ContainerStarted","Data":"ea2366595d97abc79dba6c67e610750806634f83af58dd2f81041324e739482d"} Jul 21 14:48:41.106551 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:41.106462 2570 generic.go:356] "Generic (PLEG): container finished" podID="95b180ec-0fe8-4af2-b9a3-81d2216ad865" containerID="e81255af34432caf5074273adbe1e4f965f837521cc64d7d4c444c8ba1752fd8" exitCode=0 Jul 21 14:48:41.106949 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:41.106548 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" event={"ID":"95b180ec-0fe8-4af2-b9a3-81d2216ad865","Type":"ContainerDied","Data":"e81255af34432caf5074273adbe1e4f965f837521cc64d7d4c444c8ba1752fd8"} Jul 21 14:48:42.269686 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:42.269662 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:42.328916 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:42.328892 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b180ec-0fe8-4af2-b9a3-81d2216ad865-maas-api-service-ca\") pod \"95b180ec-0fe8-4af2-b9a3-81d2216ad865\" (UID: \"95b180ec-0fe8-4af2-b9a3-81d2216ad865\") " Jul 21 14:48:42.329252 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:42.329232 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b180ec-0fe8-4af2-b9a3-81d2216ad865-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95b180ec-0fe8-4af2-b9a3-81d2216ad865" (UID: "95b180ec-0fe8-4af2-b9a3-81d2216ad865"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:48:42.430072 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:42.430050 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95b180ec-0fe8-4af2-b9a3-81d2216ad865-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:48:43.120520 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:43.120495 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" Jul 21 14:48:43.120694 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:43.120493 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj" event={"ID":"95b180ec-0fe8-4af2-b9a3-81d2216ad865","Type":"ContainerDied","Data":"ea2366595d97abc79dba6c67e610750806634f83af58dd2f81041324e739482d"} Jul 21 14:48:43.120694 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:43.120651 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea2366595d97abc79dba6c67e610750806634f83af58dd2f81041324e739482d" Jul 21 14:48:50.148580 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:50.148548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:54.592766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:54.592726 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:54.593158 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:54.593101 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj"] Jul 21 14:48:54.595805 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:54.595784 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e24b8b08-wfjnj"] Jul 21 14:48:54.761520 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:54.761481 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:54.984465 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:54.984436 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:55.498038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:55.497991 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:55.670957 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:55.670926 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:48:56.141249 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:56.141219 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b180ec-0fe8-4af2-b9a3-81d2216ad865" path="/var/lib/kubelet/pods/95b180ec-0fe8-4af2-b9a3-81d2216ad865/volumes" Jul 21 14:48:57.141977 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:48:57.141940 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:03.718388 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:03.718357 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:03.836954 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:03.836924 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:03.931274 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:03.931236 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:04.033111 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:04.033048 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:04.437687 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:04.437661 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:30.414069 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:30.414033 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:38.147982 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:38.147949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:38.326396 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:38.326368 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:49:56.183298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:49:56.183256 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:00.632408 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:00.632369 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:00.818667 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:00.818630 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:00.997705 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:00.997672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:11.295867 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:11.295783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:12.191466 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:12.191436 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:19.716290 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:19.716259 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:19.831621 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:19.831596 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:20.642436 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:20.642408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:20.804091 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:20.804065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:21.062640 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:21.062555 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:22.391807 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:22.391778 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:30.139126 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:30.139088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:30.441410 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:30.441327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:30.963414 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:30.963367 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:37.293801 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:37.293768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:37.393203 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:37.393168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:37.852561 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:37.852532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:37.941548 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:37.941500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:38.041507 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:38.041463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:40.949144 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:40.949111 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:41.088989 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:41.088962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:41.258443 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:41.258367 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:41.765383 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:41.765353 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:41.898064 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:41.898030 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:41.997327 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:41.997303 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:44.932634 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:44.932593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:45.047855 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:45.047828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:45.411374 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:45.411323 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:46.039444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:46.039407 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:46.139243 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:46.139215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:46.459679 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:46.459651 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:53.392438 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.392405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:53.588167 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.588133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:50:53.753160 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.753083 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml"] Jul 21 14:50:53.753562 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.753544 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b180ec-0fe8-4af2-b9a3-81d2216ad865" containerName="revoke-keys" Jul 21 14:50:53.753649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.753563 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b180ec-0fe8-4af2-b9a3-81d2216ad865" containerName="revoke-keys" Jul 21 14:50:53.753708 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.753647 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b180ec-0fe8-4af2-b9a3-81d2216ad865" containerName="revoke-keys" Jul 21 14:50:53.756825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.756804 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:53.760497 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.760480 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:50:53.770482 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.770462 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml"] Jul 21 14:50:53.948051 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:53.948021 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787d989e-09bc-4388-b26a-d5006593d5e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml\" (UID: \"787d989e-09bc-4388-b26a-d5006593d5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:54.048744 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.048677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787d989e-09bc-4388-b26a-d5006593d5e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml\" (UID: \"787d989e-09bc-4388-b26a-d5006593d5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:54.049352 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.049331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787d989e-09bc-4388-b26a-d5006593d5e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml\" (UID: \"787d989e-09bc-4388-b26a-d5006593d5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:54.066752 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.066728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:54.405090 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.405068 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml"] Jul 21 14:50:54.407692 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:50:54.407664 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787d989e_09bc_4388_b26a_d5006593d5e6.slice/crio-02a117000bd6b4d8631f8c235ff735a2735e8ee1b5bd7c584f3394a6825db94d WatchSource:0}: Error finding container 02a117000bd6b4d8631f8c235ff735a2735e8ee1b5bd7c584f3394a6825db94d: Status 404 returned error can't find the container with id 02a117000bd6b4d8631f8c235ff735a2735e8ee1b5bd7c584f3394a6825db94d Jul 21 14:50:54.778198 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.778164 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" event={"ID":"787d989e-09bc-4388-b26a-d5006593d5e6","Type":"ContainerStarted","Data":"357727f1d65a68f2bb73c598cb27c3084d06c33652513690610606fec25b2790"} Jul 21 14:50:54.778328 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.778218 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" event={"ID":"787d989e-09bc-4388-b26a-d5006593d5e6","Type":"ContainerStarted","Data":"02a117000bd6b4d8631f8c235ff735a2735e8ee1b5bd7c584f3394a6825db94d"} Jul 21 14:50:54.799753 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:54.799711 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" podStartSLOduration=1.799698572 podStartE2EDuration="1.799698572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:50:54.797688106 +0000 UTC m=+2017.165198134" watchObservedRunningTime="2026-07-21 14:50:54.799698572 +0000 UTC m=+2017.167208599" Jul 21 14:50:55.784608 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:55.784576 2570 generic.go:356] "Generic (PLEG): container finished" podID="787d989e-09bc-4388-b26a-d5006593d5e6" containerID="357727f1d65a68f2bb73c598cb27c3084d06c33652513690610606fec25b2790" exitCode=0 Jul 21 14:50:55.784968 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:55.784653 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" event={"ID":"787d989e-09bc-4388-b26a-d5006593d5e6","Type":"ContainerDied","Data":"357727f1d65a68f2bb73c598cb27c3084d06c33652513690610606fec25b2790"} Jul 21 14:50:56.920396 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:56.920374 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:56.971404 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:56.971377 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787d989e-09bc-4388-b26a-d5006593d5e6-maas-api-service-ca\") pod \"787d989e-09bc-4388-b26a-d5006593d5e6\" (UID: \"787d989e-09bc-4388-b26a-d5006593d5e6\") " Jul 21 14:50:56.971678 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:56.971658 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787d989e-09bc-4388-b26a-d5006593d5e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "787d989e-09bc-4388-b26a-d5006593d5e6" (UID: "787d989e-09bc-4388-b26a-d5006593d5e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:50:57.072225 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:57.072175 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/787d989e-09bc-4388-b26a-d5006593d5e6-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:50:57.795738 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:57.795713 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" Jul 21 14:50:57.795861 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:57.795738 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml" event={"ID":"787d989e-09bc-4388-b26a-d5006593d5e6","Type":"ContainerDied","Data":"02a117000bd6b4d8631f8c235ff735a2735e8ee1b5bd7c584f3394a6825db94d"} Jul 21 14:50:57.795861 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:50:57.795783 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02a117000bd6b4d8631f8c235ff735a2735e8ee1b5bd7c584f3394a6825db94d" Jul 21 14:51:04.433945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:04.433911 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:08.861118 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:08.861083 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:08.870592 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:08.870566 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml"] Jul 21 14:51:08.873820 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:08.873799 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15f01a6b-w8sml"] Jul 21 14:51:09.763927 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:09.763897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:10.141038 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:10.140991 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787d989e-09bc-4388-b26a-d5006593d5e6" path="/var/lib/kubelet/pods/787d989e-09bc-4388-b26a-d5006593d5e6/volumes" Jul 21 14:51:15.658903 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:15.658868 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:15.760306 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:15.760268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:15.922069 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:15.921986 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:16.150611 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:16.150577 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:16.263853 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:16.263774 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:16.364186 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:16.364156 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:16.497119 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:16.497090 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:17.443485 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:17.443456 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:22.747678 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:22.747644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:22.833598 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:22.833570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:22.917901 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:22.917868 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:23.021475 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:23.021400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:23.132132 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:23.132096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:29.401762 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:29.401729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:29.500191 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:29.500163 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:36.227915 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:36.227880 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:36.500770 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:36.500688 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:36.663754 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:36.663719 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:39.256145 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:39.256076 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:39.348847 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:39.348814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:39.426963 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:39.426934 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:39.670609 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:39.670578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:40.113164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:40.113081 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:40.208373 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:40.208343 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:42.537491 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:42.537410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:42.639388 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:42.639348 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:51:43.727919 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:51:43.727886 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:52:04.691171 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:04.691131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:52:18.246407 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:18.246379 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:52:18.253077 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:18.253056 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:52:22.202822 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.202792 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2"] Jul 21 14:52:22.203370 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.203348 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787d989e-09bc-4388-b26a-d5006593d5e6" containerName="revoke-keys" Jul 21 14:52:22.203448 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.203373 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="787d989e-09bc-4388-b26a-d5006593d5e6" containerName="revoke-keys" Jul 21 14:52:22.203521 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.203509 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="787d989e-09bc-4388-b26a-d5006593d5e6" containerName="revoke-keys" Jul 21 14:52:22.206502 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.206482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:22.209141 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.209117 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:52:22.225575 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.225553 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2"] Jul 21 14:52:22.257342 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.257311 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b26a196-6ba3-4577-998a-0a58a3cdc5a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02afa090-zdvs2\" (UID: \"5b26a196-6ba3-4577-998a-0a58a3cdc5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:22.358083 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.358056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b26a196-6ba3-4577-998a-0a58a3cdc5a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02afa090-zdvs2\" (UID: \"5b26a196-6ba3-4577-998a-0a58a3cdc5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:22.358645 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.358619 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b26a196-6ba3-4577-998a-0a58a3cdc5a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02afa090-zdvs2\" (UID: \"5b26a196-6ba3-4577-998a-0a58a3cdc5a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:22.516762 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.516703 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:22.656270 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:22.656246 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2"] Jul 21 14:52:22.658432 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:52:22.658406 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b26a196_6ba3_4577_998a_0a58a3cdc5a5.slice/crio-1e233e31b5ffeb4617d829eb30ced36d9bd58a8d577384254908826a8c4491eb WatchSource:0}: Error finding container 1e233e31b5ffeb4617d829eb30ced36d9bd58a8d577384254908826a8c4491eb: Status 404 returned error can't find the container with id 1e233e31b5ffeb4617d829eb30ced36d9bd58a8d577384254908826a8c4491eb Jul 21 14:52:23.220349 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:23.220288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" event={"ID":"5b26a196-6ba3-4577-998a-0a58a3cdc5a5","Type":"ContainerStarted","Data":"646e2b5c72c281402750922f3047439a7af8db9c007cb5330784825e578025af"} Jul 21 14:52:23.220349 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:23.220335 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" event={"ID":"5b26a196-6ba3-4577-998a-0a58a3cdc5a5","Type":"ContainerStarted","Data":"1e233e31b5ffeb4617d829eb30ced36d9bd58a8d577384254908826a8c4491eb"} Jul 21 14:52:23.238853 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:23.238811 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" podStartSLOduration=1.238797924 podStartE2EDuration="1.238797924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:52:23.238034525 +0000 UTC m=+2105.605544555" watchObservedRunningTime="2026-07-21 14:52:23.238797924 +0000 UTC m=+2105.606307952" Jul 21 14:52:24.226915 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:24.226880 2570 generic.go:356] "Generic (PLEG): container finished" podID="5b26a196-6ba3-4577-998a-0a58a3cdc5a5" containerID="646e2b5c72c281402750922f3047439a7af8db9c007cb5330784825e578025af" exitCode=0 Jul 21 14:52:24.227288 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:24.226919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" event={"ID":"5b26a196-6ba3-4577-998a-0a58a3cdc5a5","Type":"ContainerDied","Data":"646e2b5c72c281402750922f3047439a7af8db9c007cb5330784825e578025af"} Jul 21 14:52:25.367282 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:25.367259 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:25.483690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:25.483628 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b26a196-6ba3-4577-998a-0a58a3cdc5a5-maas-api-service-ca\") pod \"5b26a196-6ba3-4577-998a-0a58a3cdc5a5\" (UID: \"5b26a196-6ba3-4577-998a-0a58a3cdc5a5\") " Jul 21 14:52:25.483940 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:25.483920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b26a196-6ba3-4577-998a-0a58a3cdc5a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b26a196-6ba3-4577-998a-0a58a3cdc5a5" (UID: "5b26a196-6ba3-4577-998a-0a58a3cdc5a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:52:25.584308 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:25.584284 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b26a196-6ba3-4577-998a-0a58a3cdc5a5-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:52:26.237433 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:26.237410 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" Jul 21 14:52:26.237576 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:26.237446 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2" event={"ID":"5b26a196-6ba3-4577-998a-0a58a3cdc5a5","Type":"ContainerDied","Data":"1e233e31b5ffeb4617d829eb30ced36d9bd58a8d577384254908826a8c4491eb"} Jul 21 14:52:26.237576 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:26.237488 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e233e31b5ffeb4617d829eb30ced36d9bd58a8d577384254908826a8c4491eb" Jul 21 14:52:37.925730 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:37.925684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:52:42.311658 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:42.311612 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2"] Jul 21 14:52:42.313416 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:42.313393 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02afa090-zdvs2"] Jul 21 14:52:42.317797 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:42.317760 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:52:44.146446 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:44.146408 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b26a196-6ba3-4577-998a-0a58a3cdc5a5" path="/var/lib/kubelet/pods/5b26a196-6ba3-4577-998a-0a58a3cdc5a5/volumes" Jul 21 14:52:44.478778 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:44.478669 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:52:49.549354 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:52:49.549314 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:18.321250 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:18.321163 2570 scope.go:117] "RemoveContainer" containerID="9e43bdc2afcdc1d2c7374816dd3e5a3d98418b64b945dcdc2ccf1bdd31ac37bc" Jul 21 14:53:18.330206 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:18.330182 2570 scope.go:117] "RemoveContainer" containerID="e8b11f089c81963c284be4331fa1a145c456b16b15ebe327202bdc60fed899da" Jul 21 14:53:26.045318 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:26.045264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:28.260898 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:28.260857 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:33.116532 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:33.116498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:33.212554 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:33.212523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:34.448282 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:34.448247 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:34.531935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:34.531879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:34.783690 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:34.783615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:34.984956 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:34.984922 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:36.024677 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:36.024645 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:36.237703 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:36.237675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:36.916173 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:36.916137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:37.015491 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:37.015463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:37.129423 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:37.129394 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:38.432689 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:38.432653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:44.817809 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:44.817771 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:44.900859 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:44.900829 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:45.503147 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:45.503113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:52.702333 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:52.702301 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:52.811981 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:52.811952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:53:52.905714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:53:52.905681 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:03.419948 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:03.419917 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:03.755041 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:03.754960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:18.353242 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:18.353207 2570 scope.go:117] "RemoveContainer" containerID="c6a6e7d91120d86bca5ce3471d633deeabd35a3e70bc0953ed584a80e0b3ab35" Jul 21 14:54:24.716582 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:24.716546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:31.871288 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:31.871254 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:42.178911 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:42.178836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:43.487928 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:43.487834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:43.631682 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:43.631653 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:44.076668 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:44.076638 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:44.401963 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:44.401935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:44.564217 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:44.564190 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:50.877266 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:50.877232 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:56.997750 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:56.997712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:54:57.137945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:54:57.137915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:05.480487 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:05.480456 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:05.578270 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:05.578245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:14.388647 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.388603 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn"] Jul 21 14:55:14.389115 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.389043 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b26a196-6ba3-4577-998a-0a58a3cdc5a5" containerName="revoke-keys" Jul 21 14:55:14.389115 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.389055 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b26a196-6ba3-4577-998a-0a58a3cdc5a5" containerName="revoke-keys" Jul 21 14:55:14.389115 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.389112 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b26a196-6ba3-4577-998a-0a58a3cdc5a5" containerName="revoke-keys" Jul 21 14:55:14.392246 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.392228 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.399371 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.399355 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db7687\"/\"openshift-service-ca.crt\"" Jul 21 14:55:14.401136 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.401119 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db7687\"/\"test-model-db7687-kserve-self-signed-certs\"" Jul 21 14:55:14.401227 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.401154 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db7687\"/\"kube-root-ca.crt\"" Jul 21 14:55:14.403088 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.403071 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db7687\"/\"default-dockercfg-bp76j\"" Jul 21 14:55:14.411503 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.411482 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn"] Jul 21 14:55:14.416623 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.416600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:14.487588 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.487558 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhptr\" (UniqueName: \"kubernetes.io/projected/54402d00-76a3-4043-a643-981c198d2087-kube-api-access-vhptr\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.487718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.487607 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-tmp-dir\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.487718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.487629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-model-cache\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.487718 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.487691 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-dshm\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.487844 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.487727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-home\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.487844 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.487743 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54402d00-76a3-4043-a643-981c198d2087-tls-certs\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.588636 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-tmp-dir\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.588802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588642 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-model-cache\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.588802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588683 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-dshm\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.588802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588713 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-home\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.588802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54402d00-76a3-4043-a643-981c198d2087-tls-certs\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.588802 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588780 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhptr\" (UniqueName: \"kubernetes.io/projected/54402d00-76a3-4043-a643-981c198d2087-kube-api-access-vhptr\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.589091 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.588992 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-tmp-dir\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.589151 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.589125 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-model-cache\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.589193 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.589157 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-home\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.590830 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.590810 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-dshm\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.591208 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.591192 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54402d00-76a3-4043-a643-981c198d2087-tls-certs\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.599026 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.598983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhptr\" (UniqueName: \"kubernetes.io/projected/54402d00-76a3-4043-a643-981c198d2087-kube-api-access-vhptr\") pod \"test-model-db7687-kserve-57776497d5-tmwcn\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.701091 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.701000 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:14.853745 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:14.853715 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn"] Jul 21 14:55:14.856895 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:55:14.856864 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54402d00_76a3_4043_a643_981c198d2087.slice/crio-f3cf29f42e1425529e33302a141b363416d30351895b317c418f3c0a283b00d3 WatchSource:0}: Error finding container f3cf29f42e1425529e33302a141b363416d30351895b317c418f3c0a283b00d3: Status 404 returned error can't find the container with id f3cf29f42e1425529e33302a141b363416d30351895b317c418f3c0a283b00d3 Jul 21 14:55:15.133122 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:15.133033 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" event={"ID":"54402d00-76a3-4043-a643-981c198d2087","Type":"ContainerStarted","Data":"fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8"} Jul 21 14:55:15.133122 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:15.133072 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" event={"ID":"54402d00-76a3-4043-a643-981c198d2087","Type":"ContainerStarted","Data":"f3cf29f42e1425529e33302a141b363416d30351895b317c418f3c0a283b00d3"} Jul 21 14:55:15.157714 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:15.157655 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" podStartSLOduration=1.157637152 podStartE2EDuration="1.157637152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:55:15.155679777 +0000 UTC m=+2277.523189805" watchObservedRunningTime="2026-07-21 14:55:15.157637152 +0000 UTC m=+2277.525147182" Jul 21 14:55:18.373834 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:18.373804 2570 scope.go:117] "RemoveContainer" containerID="e81255af34432caf5074273adbe1e4f965f837521cc64d7d4c444c8ba1752fd8" Jul 21 14:55:24.701884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:24.701840 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:24.702333 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:24.701904 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:24.714144 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:24.714115 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:25.195028 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:25.194984 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:55:38.105948 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:38.105921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:46.764651 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.764616 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr"] Jul 21 14:55:46.768593 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.768572 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.772706 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.772682 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6c5a8f\"/\"default-dockercfg-64nss\"" Jul 21 14:55:46.772824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.772700 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6c5a8f\"/\"test-model-6c5a8f-kserve-self-signed-certs\"" Jul 21 14:55:46.772824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.772705 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6c5a8f\"/\"openshift-service-ca.crt\"" Jul 21 14:55:46.772824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.772694 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6c5a8f\"/\"kube-root-ca.crt\"" Jul 21 14:55:46.780034 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.779990 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr"] Jul 21 14:55:46.799428 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.799406 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:46.851905 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.851880 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-model-cache\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.852043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.851921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-home\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.852043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.851941 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc1817f-40e8-4aaf-8970-47e7f327de15-tls-certs\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.852143 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.852057 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-dshm\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.852143 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.852087 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-tmp-dir\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.852143 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.852110 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4wh\" (UniqueName: \"kubernetes.io/projected/2cc1817f-40e8-4aaf-8970-47e7f327de15-kube-api-access-zq4wh\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.953653 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.953620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-model-cache\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.953825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.953669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-home\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.953825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.953685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc1817f-40e8-4aaf-8970-47e7f327de15-tls-certs\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.953825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.953729 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-dshm\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.953825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.953748 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-tmp-dir\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.953825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.953771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4wh\" (UniqueName: \"kubernetes.io/projected/2cc1817f-40e8-4aaf-8970-47e7f327de15-kube-api-access-zq4wh\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.954114 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.954095 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-model-cache\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.954168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.954127 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-home\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.954168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.954161 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-tmp-dir\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.956033 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.955982 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-dshm\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.956268 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.956251 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc1817f-40e8-4aaf-8970-47e7f327de15-tls-certs\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:46.963168 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:46.963150 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4wh\" (UniqueName: \"kubernetes.io/projected/2cc1817f-40e8-4aaf-8970-47e7f327de15-kube-api-access-zq4wh\") pod \"test-model-6c5a8f-kserve-594c7ff8b7-7r2rr\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:47.078474 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:47.078398 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:47.213435 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:47.213409 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr"] Jul 21 14:55:47.214828 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:55:47.214801 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cc1817f_40e8_4aaf_8970_47e7f327de15.slice/crio-067d2298b222aec87c8056c8021b12975893f9551d5538929e7994d8839c11fa WatchSource:0}: Error finding container 067d2298b222aec87c8056c8021b12975893f9551d5538929e7994d8839c11fa: Status 404 returned error can't find the container with id 067d2298b222aec87c8056c8021b12975893f9551d5538929e7994d8839c11fa Jul 21 14:55:47.289165 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:47.289136 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" event={"ID":"2cc1817f-40e8-4aaf-8970-47e7f327de15","Type":"ContainerStarted","Data":"7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1"} Jul 21 14:55:47.289278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:47.289179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" event={"ID":"2cc1817f-40e8-4aaf-8970-47e7f327de15","Type":"ContainerStarted","Data":"067d2298b222aec87c8056c8021b12975893f9551d5538929e7994d8839c11fa"} Jul 21 14:55:47.312146 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:47.312100 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" podStartSLOduration=1.312085291 podStartE2EDuration="1.312085291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:55:47.30952457 +0000 UTC m=+2309.677034612" watchObservedRunningTime="2026-07-21 14:55:47.312085291 +0000 UTC m=+2309.679595346" Jul 21 14:55:48.395530 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:48.395490 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:48.490235 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:48.490190 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:55:57.078631 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:57.078596 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:57.079001 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:57.078645 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:57.090717 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:57.090694 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:55:57.350661 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:55:57.350597 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:56:32.933863 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:32.933827 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn"] Jul 21 14:56:32.961916 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:32.961891 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:33.925445 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:33.925413 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr"] Jul 21 14:56:33.950744 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:33.950715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:34.067197 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.067167 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq"] Jul 21 14:56:34.070517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.070502 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:34.073569 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.073549 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:56:34.081378 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.081356 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq"] Jul 21 14:56:34.134360 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.134338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc93e599-3cb0-4717-be04-9e2f14696aad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db7687-gdchq\" (UID: \"cc93e599-3cb0-4717-be04-9e2f14696aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:34.235337 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.235277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc93e599-3cb0-4717-be04-9e2f14696aad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db7687-gdchq\" (UID: \"cc93e599-3cb0-4717-be04-9e2f14696aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:34.235860 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.235843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc93e599-3cb0-4717-be04-9e2f14696aad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-db7687-gdchq\" (UID: \"cc93e599-3cb0-4717-be04-9e2f14696aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:34.381043 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.381002 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:34.522150 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:56:34.522123 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc93e599_3cb0_4717_be04_9e2f14696aad.slice/crio-3edb106a512dfaff58c2b2ed6185fe175ca537d8e7dfc2311559b41d1e8f4ea7 WatchSource:0}: Error finding container 3edb106a512dfaff58c2b2ed6185fe175ca537d8e7dfc2311559b41d1e8f4ea7: Status 404 returned error can't find the container with id 3edb106a512dfaff58c2b2ed6185fe175ca537d8e7dfc2311559b41d1e8f4ea7 Jul 21 14:56:34.522701 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.522664 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq"] Jul 21 14:56:34.529172 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.529141 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" event={"ID":"cc93e599-3cb0-4717-be04-9e2f14696aad","Type":"ContainerStarted","Data":"3edb106a512dfaff58c2b2ed6185fe175ca537d8e7dfc2311559b41d1e8f4ea7"} Jul 21 14:56:34.815033 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.814997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:34.914973 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:34.914944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:35.535164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:35.535137 2570 generic.go:356] "Generic (PLEG): container finished" podID="cc93e599-3cb0-4717-be04-9e2f14696aad" containerID="405e51573f5c3145cf4739ceb76c911fa21fc0e3ada5238ebd781375709217f9" exitCode=0 Jul 21 14:56:35.535546 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:35.535174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" event={"ID":"cc93e599-3cb0-4717-be04-9e2f14696aad","Type":"ContainerDied","Data":"405e51573f5c3145cf4739ceb76c911fa21fc0e3ada5238ebd781375709217f9"} Jul 21 14:56:36.678093 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:36.678068 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:36.757257 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:36.757234 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc93e599-3cb0-4717-be04-9e2f14696aad-maas-api-service-ca\") pod \"cc93e599-3cb0-4717-be04-9e2f14696aad\" (UID: \"cc93e599-3cb0-4717-be04-9e2f14696aad\") " Jul 21 14:56:36.757617 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:36.757597 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc93e599-3cb0-4717-be04-9e2f14696aad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc93e599-3cb0-4717-be04-9e2f14696aad" (UID: "cc93e599-3cb0-4717-be04-9e2f14696aad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:56:36.858611 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:36.858547 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc93e599-3cb0-4717-be04-9e2f14696aad-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:37.547262 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:37.547237 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" Jul 21 14:56:37.547431 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:37.547271 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq" event={"ID":"cc93e599-3cb0-4717-be04-9e2f14696aad","Type":"ContainerDied","Data":"3edb106a512dfaff58c2b2ed6185fe175ca537d8e7dfc2311559b41d1e8f4ea7"} Jul 21 14:56:37.547431 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:37.547317 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3edb106a512dfaff58c2b2ed6185fe175ca537d8e7dfc2311559b41d1e8f4ea7" Jul 21 14:56:44.719904 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:44.719867 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:47.952925 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:47.952875 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" podUID="54402d00-76a3-4043-a643-981c198d2087" containerName="main" containerID="cri-o://fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8" gracePeriod=45 Jul 21 14:56:48.205116 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.205091 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:56:48.346624 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346541 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-tmp-dir\") pod \"54402d00-76a3-4043-a643-981c198d2087\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " Jul 21 14:56:48.346624 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346585 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-home\") pod \"54402d00-76a3-4043-a643-981c198d2087\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " Jul 21 14:56:48.346624 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346616 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-model-cache\") pod \"54402d00-76a3-4043-a643-981c198d2087\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " Jul 21 14:56:48.346869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346688 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhptr\" (UniqueName: \"kubernetes.io/projected/54402d00-76a3-4043-a643-981c198d2087-kube-api-access-vhptr\") pod \"54402d00-76a3-4043-a643-981c198d2087\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " Jul 21 14:56:48.346869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346730 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-dshm\") pod \"54402d00-76a3-4043-a643-981c198d2087\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " Jul 21 14:56:48.346869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346791 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54402d00-76a3-4043-a643-981c198d2087-tls-certs\") pod \"54402d00-76a3-4043-a643-981c198d2087\" (UID: \"54402d00-76a3-4043-a643-981c198d2087\") " Jul 21 14:56:48.346869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346801 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-home" (OuterVolumeSpecName: "home") pod "54402d00-76a3-4043-a643-981c198d2087" (UID: "54402d00-76a3-4043-a643-981c198d2087"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:48.346869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346816 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54402d00-76a3-4043-a643-981c198d2087" (UID: "54402d00-76a3-4043-a643-981c198d2087"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:48.346869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.346843 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-model-cache" (OuterVolumeSpecName: "model-cache") pod "54402d00-76a3-4043-a643-981c198d2087" (UID: "54402d00-76a3-4043-a643-981c198d2087"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:48.347175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.347090 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-tmp-dir\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:48.347175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.347108 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-home\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:48.347175 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.347121 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-model-cache\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:48.348877 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.348851 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54402d00-76a3-4043-a643-981c198d2087-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54402d00-76a3-4043-a643-981c198d2087" (UID: "54402d00-76a3-4043-a643-981c198d2087"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:56:48.348948 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.348894 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-dshm" (OuterVolumeSpecName: "dshm") pod "54402d00-76a3-4043-a643-981c198d2087" (UID: "54402d00-76a3-4043-a643-981c198d2087"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:48.348984 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.348948 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54402d00-76a3-4043-a643-981c198d2087-kube-api-access-vhptr" (OuterVolumeSpecName: "kube-api-access-vhptr") pod "54402d00-76a3-4043-a643-981c198d2087" (UID: "54402d00-76a3-4043-a643-981c198d2087"). InnerVolumeSpecName "kube-api-access-vhptr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:56:48.448391 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.448356 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54402d00-76a3-4043-a643-981c198d2087-dshm\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:48.448391 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.448387 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54402d00-76a3-4043-a643-981c198d2087-tls-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:48.448391 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.448399 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhptr\" (UniqueName: \"kubernetes.io/projected/54402d00-76a3-4043-a643-981c198d2087-kube-api-access-vhptr\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:48.609661 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.609593 2570 generic.go:356] "Generic (PLEG): container finished" podID="54402d00-76a3-4043-a643-981c198d2087" containerID="fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8" exitCode=0 Jul 21 14:56:48.609661 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.609630 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" Jul 21 14:56:48.609831 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.609632 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" event={"ID":"54402d00-76a3-4043-a643-981c198d2087","Type":"ContainerDied","Data":"fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8"} Jul 21 14:56:48.609831 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.609739 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn" event={"ID":"54402d00-76a3-4043-a643-981c198d2087","Type":"ContainerDied","Data":"f3cf29f42e1425529e33302a141b363416d30351895b317c418f3c0a283b00d3"} Jul 21 14:56:48.609831 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.609756 2570 scope.go:117] "RemoveContainer" containerID="fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8" Jul 21 14:56:48.618097 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.618064 2570 scope.go:117] "RemoveContainer" containerID="fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8" Jul 21 14:56:48.618348 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:56:48.618323 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8\": container with ID starting with fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8 not found: ID does not exist" containerID="fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8" Jul 21 14:56:48.618440 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.618354 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8"} err="failed to get container status \"fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8\": rpc error: code = NotFound desc = could not find container \"fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8\": container with ID starting with fa9aad4b259687409c442f8678ab02b610d906552513489f8cef423cf4cd6aa8 not found: ID does not exist" Jul 21 14:56:48.637736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.637713 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn"] Jul 21 14:56:48.645176 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.645154 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-db7687/test-model-db7687-kserve-57776497d5-tmwcn"] Jul 21 14:56:48.944974 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:48.944950 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" podUID="2cc1817f-40e8-4aaf-8970-47e7f327de15" containerName="main" containerID="cri-o://7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1" gracePeriod=45 Jul 21 14:56:49.166760 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.166727 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:49.185031 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.184991 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq"] Jul 21 14:56:49.187995 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.187973 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-db7687-gdchq"] Jul 21 14:56:49.201053 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.200984 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:56:49.255098 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255070 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-tmp-dir\") pod \"2cc1817f-40e8-4aaf-8970-47e7f327de15\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " Jul 21 14:56:49.255315 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255295 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc1817f-40e8-4aaf-8970-47e7f327de15-tls-certs\") pod \"2cc1817f-40e8-4aaf-8970-47e7f327de15\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " Jul 21 14:56:49.255492 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255478 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-home\") pod \"2cc1817f-40e8-4aaf-8970-47e7f327de15\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " Jul 21 14:56:49.255647 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255633 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-dshm\") pod \"2cc1817f-40e8-4aaf-8970-47e7f327de15\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " Jul 21 14:56:49.255796 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255774 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-model-cache\") pod \"2cc1817f-40e8-4aaf-8970-47e7f327de15\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " Jul 21 14:56:49.255941 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255828 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq4wh\" (UniqueName: \"kubernetes.io/projected/2cc1817f-40e8-4aaf-8970-47e7f327de15-kube-api-access-zq4wh\") pod \"2cc1817f-40e8-4aaf-8970-47e7f327de15\" (UID: \"2cc1817f-40e8-4aaf-8970-47e7f327de15\") " Jul 21 14:56:49.256469 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.255381 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cc1817f-40e8-4aaf-8970-47e7f327de15" (UID: "2cc1817f-40e8-4aaf-8970-47e7f327de15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:49.256597 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.256284 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-home" (OuterVolumeSpecName: "home") pod "2cc1817f-40e8-4aaf-8970-47e7f327de15" (UID: "2cc1817f-40e8-4aaf-8970-47e7f327de15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:49.256684 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.256475 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cc1817f-40e8-4aaf-8970-47e7f327de15" (UID: "2cc1817f-40e8-4aaf-8970-47e7f327de15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:49.259852 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.259818 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-dshm" (OuterVolumeSpecName: "dshm") pod "2cc1817f-40e8-4aaf-8970-47e7f327de15" (UID: "2cc1817f-40e8-4aaf-8970-47e7f327de15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 14:56:49.260408 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.260386 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc1817f-40e8-4aaf-8970-47e7f327de15-kube-api-access-zq4wh" (OuterVolumeSpecName: "kube-api-access-zq4wh") pod "2cc1817f-40e8-4aaf-8970-47e7f327de15" (UID: "2cc1817f-40e8-4aaf-8970-47e7f327de15"). InnerVolumeSpecName "kube-api-access-zq4wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 14:56:49.260606 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.260589 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc1817f-40e8-4aaf-8970-47e7f327de15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cc1817f-40e8-4aaf-8970-47e7f327de15" (UID: "2cc1817f-40e8-4aaf-8970-47e7f327de15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 14:56:49.356762 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.356734 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-model-cache\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:49.356878 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.356766 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq4wh\" (UniqueName: \"kubernetes.io/projected/2cc1817f-40e8-4aaf-8970-47e7f327de15-kube-api-access-zq4wh\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:49.356878 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.356780 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-tmp-dir\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:49.356878 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.356792 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cc1817f-40e8-4aaf-8970-47e7f327de15-tls-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:49.356878 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.356804 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-home\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:49.356878 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.356814 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cc1817f-40e8-4aaf-8970-47e7f327de15-dshm\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:56:49.589842 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.589767 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:49.615661 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.615630 2570 generic.go:356] "Generic (PLEG): container finished" podID="2cc1817f-40e8-4aaf-8970-47e7f327de15" containerID="7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1" exitCode=0 Jul 21 14:56:49.615766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.615668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" event={"ID":"2cc1817f-40e8-4aaf-8970-47e7f327de15","Type":"ContainerDied","Data":"7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1"} Jul 21 14:56:49.615766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.615696 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" Jul 21 14:56:49.615766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.615709 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr" event={"ID":"2cc1817f-40e8-4aaf-8970-47e7f327de15","Type":"ContainerDied","Data":"067d2298b222aec87c8056c8021b12975893f9551d5538929e7994d8839c11fa"} Jul 21 14:56:49.615766 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.615732 2570 scope.go:117] "RemoveContainer" containerID="7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1" Jul 21 14:56:49.624527 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.624508 2570 scope.go:117] "RemoveContainer" containerID="7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1" Jul 21 14:56:49.624757 ip-10-0-134-49 kubenswrapper[2570]: E0721 14:56:49.624740 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1\": container with ID starting with 7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1 not found: ID does not exist" containerID="7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1" Jul 21 14:56:49.624816 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.624766 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1"} err="failed to get container status \"7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1\": rpc error: code = NotFound desc = could not find container \"7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1\": container with ID starting with 7bc063a5b566f794a1f52d8a12b2449fd3de743812f81c4b8d9e575fde404ad1 not found: ID does not exist" Jul 21 14:56:49.639578 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.639554 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr"] Jul 21 14:56:49.649722 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:49.649700 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6c5a8f/test-model-6c5a8f-kserve-594c7ff8b7-7r2rr"] Jul 21 14:56:50.142180 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:50.142152 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc1817f-40e8-4aaf-8970-47e7f327de15" path="/var/lib/kubelet/pods/2cc1817f-40e8-4aaf-8970-47e7f327de15/volumes" Jul 21 14:56:50.142524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:50.142511 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54402d00-76a3-4043-a643-981c198d2087" path="/var/lib/kubelet/pods/54402d00-76a3-4043-a643-981c198d2087/volumes" Jul 21 14:56:50.142807 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:50.142795 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc93e599-3cb0-4717-be04-9e2f14696aad" path="/var/lib/kubelet/pods/cc93e599-3cb0-4717-be04-9e2f14696aad/volumes" Jul 21 14:56:56.633604 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:56.633568 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:56.784869 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:56.784834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:56.923795 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:56.923763 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:57.157643 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.157612 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:57.246283 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246222 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj"] Jul 21 14:56:57.246607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246594 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc93e599-3cb0-4717-be04-9e2f14696aad" containerName="revoke-keys" Jul 21 14:56:57.246607 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246607 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc93e599-3cb0-4717-be04-9e2f14696aad" containerName="revoke-keys" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246618 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54402d00-76a3-4043-a643-981c198d2087" containerName="main" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246624 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="54402d00-76a3-4043-a643-981c198d2087" containerName="main" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246640 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc1817f-40e8-4aaf-8970-47e7f327de15" containerName="main" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246646 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc1817f-40e8-4aaf-8970-47e7f327de15" containerName="main" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246716 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc93e599-3cb0-4717-be04-9e2f14696aad" containerName="revoke-keys" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246725 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="54402d00-76a3-4043-a643-981c198d2087" containerName="main" Jul 21 14:56:57.246736 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.246733 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc1817f-40e8-4aaf-8970-47e7f327de15" containerName="main" Jul 21 14:56:57.249791 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.249770 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:56:57.254295 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.254276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:56:57.261164 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.261140 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj"] Jul 21 14:56:57.315960 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.315930 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5dd3c2-241d-4444-b904-d6874995fd6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj\" (UID: \"da5dd3c2-241d-4444-b904-d6874995fd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:56:57.416546 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.416520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5dd3c2-241d-4444-b904-d6874995fd6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj\" (UID: \"da5dd3c2-241d-4444-b904-d6874995fd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:56:57.417117 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.417096 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5dd3c2-241d-4444-b904-d6874995fd6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj\" (UID: \"da5dd3c2-241d-4444-b904-d6874995fd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:56:57.560998 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.560934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:56:57.704193 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:57.704171 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj"] Jul 21 14:56:57.706470 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:56:57.706447 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5dd3c2_241d_4444_b904_d6874995fd6a.slice/crio-e61545ec44304ff15073277d894b9557ff017b0e0f0795be431b39892007acc9 WatchSource:0}: Error finding container e61545ec44304ff15073277d894b9557ff017b0e0f0795be431b39892007acc9: Status 404 returned error can't find the container with id e61545ec44304ff15073277d894b9557ff017b0e0f0795be431b39892007acc9 Jul 21 14:56:58.130420 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:58.130345 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:58.232153 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:58.232127 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:56:58.665433 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:58.665401 2570 generic.go:356] "Generic (PLEG): container finished" podID="da5dd3c2-241d-4444-b904-d6874995fd6a" containerID="a7a3e8b0035ffee022dcf54fae82b90a466c5d2c7f9e6c3c83548eb3d92ed7cd" exitCode=0 Jul 21 14:56:58.665614 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:58.665486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" event={"ID":"da5dd3c2-241d-4444-b904-d6874995fd6a","Type":"ContainerDied","Data":"a7a3e8b0035ffee022dcf54fae82b90a466c5d2c7f9e6c3c83548eb3d92ed7cd"} Jul 21 14:56:58.665614 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:58.665542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" event={"ID":"da5dd3c2-241d-4444-b904-d6874995fd6a","Type":"ContainerStarted","Data":"e61545ec44304ff15073277d894b9557ff017b0e0f0795be431b39892007acc9"} Jul 21 14:56:59.803565 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:59.803544 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:56:59.936928 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:59.936866 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5dd3c2-241d-4444-b904-d6874995fd6a-maas-api-service-ca\") pod \"da5dd3c2-241d-4444-b904-d6874995fd6a\" (UID: \"da5dd3c2-241d-4444-b904-d6874995fd6a\") " Jul 21 14:56:59.937233 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:56:59.937209 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5dd3c2-241d-4444-b904-d6874995fd6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da5dd3c2-241d-4444-b904-d6874995fd6a" (UID: "da5dd3c2-241d-4444-b904-d6874995fd6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:57:00.038240 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:00.038219 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da5dd3c2-241d-4444-b904-d6874995fd6a-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:57:00.677059 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:00.677034 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" Jul 21 14:57:00.677205 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:00.677059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj" event={"ID":"da5dd3c2-241d-4444-b904-d6874995fd6a","Type":"ContainerDied","Data":"e61545ec44304ff15073277d894b9557ff017b0e0f0795be431b39892007acc9"} Jul 21 14:57:00.677205 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:00.677110 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61545ec44304ff15073277d894b9557ff017b0e0f0795be431b39892007acc9" Jul 21 14:57:07.875924 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:07.875888 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:12.344938 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:12.344903 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:12.357846 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:12.357784 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj"] Jul 21 14:57:12.361292 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:12.361257 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6c5a8f-gzdkj"] Jul 21 14:57:12.716237 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:12.716202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:14.141232 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:14.141197 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5dd3c2-241d-4444-b904-d6874995fd6a" path="/var/lib/kubelet/pods/da5dd3c2-241d-4444-b904-d6874995fd6a/volumes" Jul 21 14:57:18.277804 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:18.277778 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:57:18.286140 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:18.286120 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 14:57:18.399769 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:18.399739 2570 scope.go:117] "RemoveContainer" containerID="357727f1d65a68f2bb73c598cb27c3084d06c33652513690610606fec25b2790" Jul 21 14:57:19.254247 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:19.254208 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:19.355253 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:19.355215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:19.510629 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:19.510544 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:19.751595 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:19.751567 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:20.100596 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:20.100566 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:20.417673 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:20.417642 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:20.578528 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:20.578498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:23.179292 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:23.179259 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:23.288729 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:23.288700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:23.381138 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:23.381098 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:23.480844 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:23.480779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:24.020676 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:24.020645 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:24.120276 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:24.120248 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:27.646557 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:27.646524 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:27.723843 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:27.723812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:28.120454 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:28.120384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:28.447679 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:28.447648 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:28.575388 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:28.575354 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:31.264552 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:31.264517 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:31.364305 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:31.364258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:31.449577 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:31.449546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:31.569210 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:31.569135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:32.015085 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:32.015054 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:32.116884 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:32.116855 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:35.721207 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:35.721171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:43.618377 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.618301 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts"] Jul 21 14:57:43.618746 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.618663 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5dd3c2-241d-4444-b904-d6874995fd6a" containerName="revoke-keys" Jul 21 14:57:43.618746 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.618688 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5dd3c2-241d-4444-b904-d6874995fd6a" containerName="revoke-keys" Jul 21 14:57:43.618816 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.618764 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5dd3c2-241d-4444-b904-d6874995fd6a" containerName="revoke-keys" Jul 21 14:57:43.621910 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.621895 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:43.624448 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.624422 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:57:43.629992 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.629967 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts"] Jul 21 14:57:43.686704 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.686666 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b228d0-6469-4632-8874-4ecae883a2c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts\" (UID: \"a2b228d0-6469-4632-8874-4ecae883a2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:43.787914 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.787880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b228d0-6469-4632-8874-4ecae883a2c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts\" (UID: \"a2b228d0-6469-4632-8874-4ecae883a2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:43.788503 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.788482 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b228d0-6469-4632-8874-4ecae883a2c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts\" (UID: \"a2b228d0-6469-4632-8874-4ecae883a2c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:43.934119 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:43.934097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:44.272824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:44.272727 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts"] Jul 21 14:57:44.276067 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:57:44.276040 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b228d0_6469_4632_8874_4ecae883a2c2.slice/crio-cec6bb61fb1bd62d0c0d94fdd3dab2eb31349cd83cbb8ae10963540e2555db84 WatchSource:0}: Error finding container cec6bb61fb1bd62d0c0d94fdd3dab2eb31349cd83cbb8ae10963540e2555db84: Status 404 returned error can't find the container with id cec6bb61fb1bd62d0c0d94fdd3dab2eb31349cd83cbb8ae10963540e2555db84 Jul 21 14:57:44.364054 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:44.363998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:44.463875 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:44.463854 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:44.896497 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:44.896471 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" event={"ID":"a2b228d0-6469-4632-8874-4ecae883a2c2","Type":"ContainerStarted","Data":"1d5ba565c0882fc3e31b3d1e406ae59dcf84db6d2b8f087e76308448a2590fdc"} Jul 21 14:57:44.896842 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:44.896518 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" event={"ID":"a2b228d0-6469-4632-8874-4ecae883a2c2","Type":"ContainerStarted","Data":"cec6bb61fb1bd62d0c0d94fdd3dab2eb31349cd83cbb8ae10963540e2555db84"} Jul 21 14:57:44.915461 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:44.915419 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" podStartSLOduration=1.9154050329999999 podStartE2EDuration="1.915405033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:57:44.913519921 +0000 UTC m=+2427.281029949" watchObservedRunningTime="2026-07-21 14:57:44.915405033 +0000 UTC m=+2427.282915061" Jul 21 14:57:45.903423 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:45.903389 2570 generic.go:356] "Generic (PLEG): container finished" podID="a2b228d0-6469-4632-8874-4ecae883a2c2" containerID="1d5ba565c0882fc3e31b3d1e406ae59dcf84db6d2b8f087e76308448a2590fdc" exitCode=0 Jul 21 14:57:45.903776 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:45.903476 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" event={"ID":"a2b228d0-6469-4632-8874-4ecae883a2c2","Type":"ContainerDied","Data":"1d5ba565c0882fc3e31b3d1e406ae59dcf84db6d2b8f087e76308448a2590fdc"} Jul 21 14:57:47.067055 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.067030 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:47.134037 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.133990 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b228d0-6469-4632-8874-4ecae883a2c2-maas-api-service-ca\") pod \"a2b228d0-6469-4632-8874-4ecae883a2c2\" (UID: \"a2b228d0-6469-4632-8874-4ecae883a2c2\") " Jul 21 14:57:47.134294 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.134273 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b228d0-6469-4632-8874-4ecae883a2c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2b228d0-6469-4632-8874-4ecae883a2c2" (UID: "a2b228d0-6469-4632-8874-4ecae883a2c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:57:47.235430 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.235375 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b228d0-6469-4632-8874-4ecae883a2c2-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:57:47.915821 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.915792 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" Jul 21 14:57:47.915945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.915832 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts" event={"ID":"a2b228d0-6469-4632-8874-4ecae883a2c2","Type":"ContainerDied","Data":"cec6bb61fb1bd62d0c0d94fdd3dab2eb31349cd83cbb8ae10963540e2555db84"} Jul 21 14:57:47.915945 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:47.915893 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec6bb61fb1bd62d0c0d94fdd3dab2eb31349cd83cbb8ae10963540e2555db84" Jul 21 14:57:52.929529 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:52.929496 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:53.032366 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:53.032323 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:57:58.879824 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:57:58.879791 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:03.726750 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:03.726712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:03.731833 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:03.731809 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts"] Jul 21 14:58:03.735278 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:03.735254 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6ecdf5-xbrts"] Jul 21 14:58:04.121913 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:04.121828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:04.140864 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:04.140839 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b228d0-6469-4632-8874-4ecae883a2c2" path="/var/lib/kubelet/pods/a2b228d0-6469-4632-8874-4ecae883a2c2/volumes" Jul 21 14:58:10.605193 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:10.605145 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:10.704685 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:10.704650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:10.824841 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:10.824809 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:11.076338 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.076302 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:11.148444 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.148417 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh"] Jul 21 14:58:11.148794 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.148782 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b228d0-6469-4632-8874-4ecae883a2c2" containerName="revoke-keys" Jul 21 14:58:11.148837 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.148796 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b228d0-6469-4632-8874-4ecae883a2c2" containerName="revoke-keys" Jul 21 14:58:11.148882 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.148874 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b228d0-6469-4632-8874-4ecae883a2c2" containerName="revoke-keys" Jul 21 14:58:11.151943 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.151928 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:11.154713 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.154690 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 14:58:11.166156 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.166134 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh"] Jul 21 14:58:11.318941 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.318909 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6cd2cf-c850-4355-b14e-b2230ef8e53e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63bc89-5mflh\" (UID: \"3f6cd2cf-c850-4355-b14e-b2230ef8e53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:11.419594 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.419566 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6cd2cf-c850-4355-b14e-b2230ef8e53e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63bc89-5mflh\" (UID: \"3f6cd2cf-c850-4355-b14e-b2230ef8e53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:11.420157 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.420140 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6cd2cf-c850-4355-b14e-b2230ef8e53e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-63bc89-5mflh\" (UID: \"3f6cd2cf-c850-4355-b14e-b2230ef8e53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:11.462755 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.462726 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:11.813743 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:11.813422 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh"] Jul 21 14:58:12.045689 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:12.045661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" event={"ID":"3f6cd2cf-c850-4355-b14e-b2230ef8e53e","Type":"ContainerStarted","Data":"2796058c7732156f4382cf7a9d6dde44b82d2c16ff70d0de55b54daba5e1ef7b"} Jul 21 14:58:13.051935 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:13.051897 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f6cd2cf-c850-4355-b14e-b2230ef8e53e" containerID="5027e040aaf22121d4a6ae5c5283277ad1d7579658956ad30e3f5b2ac2e4f2a4" exitCode=0 Jul 21 14:58:13.052315 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:13.051981 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" event={"ID":"3f6cd2cf-c850-4355-b14e-b2230ef8e53e","Type":"ContainerDied","Data":"5027e040aaf22121d4a6ae5c5283277ad1d7579658956ad30e3f5b2ac2e4f2a4"} Jul 21 14:58:14.190160 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:14.190137 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:14.242502 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:14.242482 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6cd2cf-c850-4355-b14e-b2230ef8e53e-maas-api-service-ca\") pod \"3f6cd2cf-c850-4355-b14e-b2230ef8e53e\" (UID: \"3f6cd2cf-c850-4355-b14e-b2230ef8e53e\") " Jul 21 14:58:14.242781 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:14.242760 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6cd2cf-c850-4355-b14e-b2230ef8e53e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f6cd2cf-c850-4355-b14e-b2230ef8e53e" (UID: "3f6cd2cf-c850-4355-b14e-b2230ef8e53e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 14:58:14.343524 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:14.343462 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f6cd2cf-c850-4355-b14e-b2230ef8e53e-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 14:58:15.062717 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:15.062691 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" Jul 21 14:58:15.062911 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:15.062734 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh" event={"ID":"3f6cd2cf-c850-4355-b14e-b2230ef8e53e","Type":"ContainerDied","Data":"2796058c7732156f4382cf7a9d6dde44b82d2c16ff70d0de55b54daba5e1ef7b"} Jul 21 14:58:15.062911 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:15.062777 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2796058c7732156f4382cf7a9d6dde44b82d2c16ff70d0de55b54daba5e1ef7b" Jul 21 14:58:26.884517 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:26.884484 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:31.267870 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:31.267839 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh"] Jul 21 14:58:31.272844 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:31.272822 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-63bc89-5mflh"] Jul 21 14:58:31.300410 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:31.300384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:31.659806 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:31.659773 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:32.141421 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:32.141392 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f6cd2cf-c850-4355-b14e-b2230ef8e53e" path="/var/lib/kubelet/pods/3f6cd2cf-c850-4355-b14e-b2230ef8e53e/volumes" Jul 21 14:58:34.166188 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:34.166141 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:34.290171 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:34.290140 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:38.828269 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:38.828234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:38.995283 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:38.995252 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:39.164388 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:39.164356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:39.365877 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:39.365844 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:39.734679 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:39.734646 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:40.089369 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:40.089288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:40.315292 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:40.315263 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:42.745229 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:42.745192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:42.873116 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:42.873089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:43.094182 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:43.094104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:43.326331 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:43.326297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:43.831584 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:43.831541 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:43.931982 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:43.931948 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:47.234362 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:47.234326 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:47.577370 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:47.577286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:47.927201 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:47.927172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:48.173845 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:48.173814 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:50.864457 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:50.864421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:50.973825 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:50.973795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:51.095649 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:51.095611 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:51.437655 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:51.437625 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:51.789511 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:51.789441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:51.986465 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:51.986434 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:58:55.369125 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:58:55.369084 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:04.768296 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:04.768258 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:04.868615 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:04.868580 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:13.094710 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:13.094627 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:13.187744 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:13.187716 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:18.436565 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.436536 2570 scope.go:117] "RemoveContainer" containerID="646e2b5c72c281402750922f3047439a7af8db9c007cb5330784825e578025af" Jul 21 14:59:18.784722 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.784643 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64"] Jul 21 14:59:18.785161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.785126 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f6cd2cf-c850-4355-b14e-b2230ef8e53e" containerName="revoke-keys" Jul 21 14:59:18.785161 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.785147 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f6cd2cf-c850-4355-b14e-b2230ef8e53e" containerName="revoke-keys" Jul 21 14:59:18.785333 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.785237 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f6cd2cf-c850-4355-b14e-b2230ef8e53e" containerName="revoke-keys" Jul 21 14:59:18.789700 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.789679 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:18.793489 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.793462 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d89949\"/\"kube-root-ca.crt\"" Jul 21 14:59:18.793610 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.793465 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d89949\"/\"default-dockercfg-9cmfc\"" Jul 21 14:59:18.793610 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.793474 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d89949\"/\"openshift-service-ca.crt\"" Jul 21 14:59:18.793610 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.793474 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d89949\"/\"auth-test-model-d89949-kserve-self-signed-certs\"" Jul 21 14:59:18.808134 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.808112 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64"] Jul 21 14:59:18.810883 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.810864 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:18.937520 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.937492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tls-certs\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:18.937644 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.937522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdtq\" (UniqueName: \"kubernetes.io/projected/e0ffe0c8-4224-4111-b0d2-d34e611aca09-kube-api-access-grdtq\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:18.937644 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.937543 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-model-cache\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:18.937644 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.937617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-home\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:18.937764 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.937652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tmp-dir\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:18.937764 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:18.937670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-dshm\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039031 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.038936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tls-certs\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039031 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.038975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdtq\" (UniqueName: \"kubernetes.io/projected/e0ffe0c8-4224-4111-b0d2-d34e611aca09-kube-api-access-grdtq\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039031 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039001 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-model-cache\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-home\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tmp-dir\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039298 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039186 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-dshm\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039539 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039499 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-model-cache\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039654 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039514 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tmp-dir\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.039654 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.039565 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-home\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.041280 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.041251 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-dshm\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.041431 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.041413 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tls-certs\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.052782 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.052761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdtq\" (UniqueName: \"kubernetes.io/projected/e0ffe0c8-4224-4111-b0d2-d34e611aca09-kube-api-access-grdtq\") pod \"auth-test-model-d89949-kserve-65ccbf67f9-r7v64\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.098805 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.098783 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:19.244943 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.244918 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64"] Jul 21 14:59:19.246439 ip-10-0-134-49 kubenswrapper[2570]: W0721 14:59:19.246412 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0ffe0c8_4224_4111_b0d2_d34e611aca09.slice/crio-d3f78d8aac451ae389b549881816c190818ac34118520e47d2e746c52038b9b6 WatchSource:0}: Error finding container d3f78d8aac451ae389b549881816c190818ac34118520e47d2e746c52038b9b6: Status 404 returned error can't find the container with id d3f78d8aac451ae389b549881816c190818ac34118520e47d2e746c52038b9b6 Jul 21 14:59:19.399355 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.399316 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" event={"ID":"e0ffe0c8-4224-4111-b0d2-d34e611aca09","Type":"ContainerStarted","Data":"b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc"} Jul 21 14:59:19.399511 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.399365 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" event={"ID":"e0ffe0c8-4224-4111-b0d2-d34e611aca09","Type":"ContainerStarted","Data":"d3f78d8aac451ae389b549881816c190818ac34118520e47d2e746c52038b9b6"} Jul 21 14:59:19.428059 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:19.427999 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" podStartSLOduration=1.42798504 podStartE2EDuration="1.42798504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 14:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 14:59:19.426395925 +0000 UTC m=+2521.793905952" watchObservedRunningTime="2026-07-21 14:59:19.42798504 +0000 UTC m=+2521.795495068" Jul 21 14:59:29.099699 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:29.099656 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:29.100234 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:29.099715 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:29.111924 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:29.111891 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:29.458021 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:29.457975 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 14:59:32.840625 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:32.840593 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:35.693791 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:35.693751 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:35.779669 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:35.779639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 14:59:35.870468 ip-10-0-134-49 kubenswrapper[2570]: I0721 14:59:35.870438 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:00:00.170093 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.170064 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l"] Jul 21 15:00:00.173516 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.173494 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" Jul 21 15:00:00.176655 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.176635 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 15:00:00.177346 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.177324 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q"] Jul 21 15:00:00.180483 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.180455 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc"] Jul 21 15:00:00.180589 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.180541 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" Jul 21 15:00:00.183789 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.183773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" Jul 21 15:00:00.184209 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.184196 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" Jul 21 15:00:00.188888 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.188865 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l"] Jul 21 15:00:00.195795 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.195776 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" Jul 21 15:00:00.207233 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.207154 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc"] Jul 21 15:00:00.237315 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.237293 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q"] Jul 21 15:00:00.241891 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.241864 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" Jul 21 15:00:00.423711 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.423685 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc"] Jul 21 15:00:00.425715 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:00:00.425690 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2129b578_d97d_4b05_a0ad_8fdfbeb8d440.slice/crio-2611be4d5fe033ed56105d0956c87cfc8ef0368bda76a42fc03cfb068d4b7c72 WatchSource:0}: Error finding container 2611be4d5fe033ed56105d0956c87cfc8ef0368bda76a42fc03cfb068d4b7c72: Status 404 returned error can't find the container with id 2611be4d5fe033ed56105d0956c87cfc8ef0368bda76a42fc03cfb068d4b7c72 Jul 21 15:00:00.565886 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.565858 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l"] Jul 21 15:00:00.570116 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:00:00.570093 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed9c11a_3b15_48ad_a824_dbdf9dde7715.slice/crio-d52dfb15004bcb00c4c4d0d42a622f287bdfe20ebe7d71d937cf85c751caea2d WatchSource:0}: Error finding container d52dfb15004bcb00c4c4d0d42a622f287bdfe20ebe7d71d937cf85c751caea2d: Status 404 returned error can't find the container with id d52dfb15004bcb00c4c4d0d42a622f287bdfe20ebe7d71d937cf85c751caea2d Jul 21 15:00:00.597693 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.597669 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q"] Jul 21 15:00:00.601097 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:00:00.601076 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae0ee80_1772_4663_9ec1_aed5e1d2e202.slice/crio-347f105011e1af32948a7b028d10d287838b3d701deddb0cb521cdf1aad7b3db WatchSource:0}: Error finding container 347f105011e1af32948a7b028d10d287838b3d701deddb0cb521cdf1aad7b3db: Status 404 returned error can't find the container with id 347f105011e1af32948a7b028d10d287838b3d701deddb0cb521cdf1aad7b3db Jul 21 15:00:00.604284 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.604260 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" event={"ID":"2ed9c11a-3b15-48ad-a824-dbdf9dde7715","Type":"ContainerStarted","Data":"d52dfb15004bcb00c4c4d0d42a622f287bdfe20ebe7d71d937cf85c751caea2d"} Jul 21 15:00:00.605349 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:00.605328 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" event={"ID":"2129b578-d97d-4b05-a0ad-8fdfbeb8d440","Type":"ContainerStarted","Data":"2611be4d5fe033ed56105d0956c87cfc8ef0368bda76a42fc03cfb068d4b7c72"} Jul 21 15:00:01.611821 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.611791 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" event={"ID":"2ed9c11a-3b15-48ad-a824-dbdf9dde7715","Type":"ContainerStarted","Data":"dc835ca758a2d60e9fc44098f5bbd392780f3c566953d4aebc3472f550101fe5"} Jul 21 15:00:01.613199 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.613173 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ae0ee80-1772-4663-9ec1-aed5e1d2e202" containerID="479871b2f65169f7f8199ac856d14d4c8e4c4edbe5d0e6f1827c404c0e201308" exitCode=0 Jul 21 15:00:01.613281 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.613248 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" event={"ID":"7ae0ee80-1772-4663-9ec1-aed5e1d2e202","Type":"ContainerDied","Data":"479871b2f65169f7f8199ac856d14d4c8e4c4edbe5d0e6f1827c404c0e201308"} Jul 21 15:00:01.613341 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.613289 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" event={"ID":"7ae0ee80-1772-4663-9ec1-aed5e1d2e202","Type":"ContainerStarted","Data":"347f105011e1af32948a7b028d10d287838b3d701deddb0cb521cdf1aad7b3db"} Jul 21 15:00:01.614681 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.614658 2570 generic.go:356] "Generic (PLEG): container finished" podID="2129b578-d97d-4b05-a0ad-8fdfbeb8d440" containerID="9494aba8a20d2155fd146fec89483cdb11210f214acece1c4bb98b4a2f9a520a" exitCode=0 Jul 21 15:00:01.614758 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.614689 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" event={"ID":"2129b578-d97d-4b05-a0ad-8fdfbeb8d440","Type":"ContainerDied","Data":"9494aba8a20d2155fd146fec89483cdb11210f214acece1c4bb98b4a2f9a520a"} Jul 21 15:00:01.638909 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:01.638869 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" podStartSLOduration=1.638858162 podStartE2EDuration="1.638858162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:00:01.638099798 +0000 UTC m=+2564.005609824" watchObservedRunningTime="2026-07-21 15:00:01.638858162 +0000 UTC m=+2564.006368229" Jul 21 15:00:02.621784 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:02.621752 2570 generic.go:356] "Generic (PLEG): container finished" podID="2ed9c11a-3b15-48ad-a824-dbdf9dde7715" containerID="dc835ca758a2d60e9fc44098f5bbd392780f3c566953d4aebc3472f550101fe5" exitCode=0 Jul 21 15:00:02.622163 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:02.621845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" event={"ID":"2ed9c11a-3b15-48ad-a824-dbdf9dde7715","Type":"ContainerDied","Data":"dc835ca758a2d60e9fc44098f5bbd392780f3c566953d4aebc3472f550101fe5"} Jul 21 15:00:02.778855 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:02.778831 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" Jul 21 15:00:02.781679 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:02.781660 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" Jul 21 15:00:03.627723 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.627695 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" Jul 21 15:00:03.627723 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.627709 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q" event={"ID":"7ae0ee80-1772-4663-9ec1-aed5e1d2e202","Type":"ContainerDied","Data":"347f105011e1af32948a7b028d10d287838b3d701deddb0cb521cdf1aad7b3db"} Jul 21 15:00:03.628321 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.627755 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347f105011e1af32948a7b028d10d287838b3d701deddb0cb521cdf1aad7b3db" Jul 21 15:00:03.629289 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.629272 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" Jul 21 15:00:03.629409 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.629321 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc" event={"ID":"2129b578-d97d-4b05-a0ad-8fdfbeb8d440","Type":"ContainerDied","Data":"2611be4d5fe033ed56105d0956c87cfc8ef0368bda76a42fc03cfb068d4b7c72"} Jul 21 15:00:03.629409 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.629374 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2611be4d5fe033ed56105d0956c87cfc8ef0368bda76a42fc03cfb068d4b7c72" Jul 21 15:00:03.766350 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:03.766325 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" Jul 21 15:00:04.635967 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:04.635898 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" event={"ID":"2ed9c11a-3b15-48ad-a824-dbdf9dde7715","Type":"ContainerDied","Data":"d52dfb15004bcb00c4c4d0d42a622f287bdfe20ebe7d71d937cf85c751caea2d"} Jul 21 15:00:04.635967 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:04.635922 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744100-vc57l" Jul 21 15:00:04.635967 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:00:04.635943 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d52dfb15004bcb00c4c4d0d42a622f287bdfe20ebe7d71d937cf85c751caea2d" Jul 21 15:01:06.095834 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:06.095800 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:11.136133 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:11.136067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:31.742267 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:31.742234 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64"] Jul 21 15:01:31.786858 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:31.786824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:32.781929 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:32.781890 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:41.470742 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.470704 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g"] Jul 21 15:01:41.478083 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478052 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed9c11a-3b15-48ad-a824-dbdf9dde7715" containerName="cleanup" Jul 21 15:01:41.478083 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478083 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed9c11a-3b15-48ad-a824-dbdf9dde7715" containerName="cleanup" Jul 21 15:01:41.478264 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478150 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2129b578-d97d-4b05-a0ad-8fdfbeb8d440" containerName="cleanup" Jul 21 15:01:41.478305 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478286 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2129b578-d97d-4b05-a0ad-8fdfbeb8d440" containerName="cleanup" Jul 21 15:01:41.478305 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478302 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae0ee80-1772-4663-9ec1-aed5e1d2e202" containerName="cleanup" Jul 21 15:01:41.478370 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478308 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae0ee80-1772-4663-9ec1-aed5e1d2e202" containerName="cleanup" Jul 21 15:01:41.478409 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478394 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2129b578-d97d-4b05-a0ad-8fdfbeb8d440" containerName="cleanup" Jul 21 15:01:41.478409 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478405 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae0ee80-1772-4663-9ec1-aed5e1d2e202" containerName="cleanup" Jul 21 15:01:41.478492 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.478411 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed9c11a-3b15-48ad-a824-dbdf9dde7715" containerName="cleanup" Jul 21 15:01:41.482707 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.482689 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.486477 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.486428 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d89949\"/\"sub-test-model-d89949-kserve-self-signed-certs\"" Jul 21 15:01:41.492833 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.491794 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g"] Jul 21 15:01:41.497025 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.496979 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:01:41.593692 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.593670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-dshm\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.593799 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.593700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppg4\" (UniqueName: \"kubernetes.io/projected/fcedfb22-805c-4a1c-a376-4244be61a94d-kube-api-access-mppg4\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.593799 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.593723 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-home\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.593890 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.593823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-model-cache\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.593890 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.593847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcedfb22-805c-4a1c-a376-4244be61a94d-tls-certs\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.593890 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.593882 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-tmp-dir\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694278 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-dshm\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694398 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppg4\" (UniqueName: \"kubernetes.io/projected/fcedfb22-805c-4a1c-a376-4244be61a94d-kube-api-access-mppg4\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694398 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-home\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694398 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694348 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-model-cache\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694398 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcedfb22-805c-4a1c-a376-4244be61a94d-tls-certs\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694398 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-tmp-dir\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694749 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694726 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-home\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694822 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-tmp-dir\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.694882 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.694845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-model-cache\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.696535 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.696508 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-dshm\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.696810 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.696792 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcedfb22-805c-4a1c-a376-4244be61a94d-tls-certs\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.706694 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.706671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppg4\" (UniqueName: \"kubernetes.io/projected/fcedfb22-805c-4a1c-a376-4244be61a94d-kube-api-access-mppg4\") pod \"sub-test-model-d89949-kserve-5c49b977cc-txg7g\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.794099 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.794040 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:41.929345 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:41.929323 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g"] Jul 21 15:01:41.931450 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:01:41.931425 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcedfb22_805c_4a1c_a376_4244be61a94d.slice/crio-bb63bebb33366dc559c765e0d7572ace2d055dae5af1abbabf669b993bbbc2f8 WatchSource:0}: Error finding container bb63bebb33366dc559c765e0d7572ace2d055dae5af1abbabf669b993bbbc2f8: Status 404 returned error can't find the container with id bb63bebb33366dc559c765e0d7572ace2d055dae5af1abbabf669b993bbbc2f8 Jul 21 15:01:42.149230 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:42.149199 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" event={"ID":"fcedfb22-805c-4a1c-a376-4244be61a94d","Type":"ContainerStarted","Data":"ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c"} Jul 21 15:01:42.149370 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:42.149237 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" event={"ID":"fcedfb22-805c-4a1c-a376-4244be61a94d","Type":"ContainerStarted","Data":"bb63bebb33366dc559c765e0d7572ace2d055dae5af1abbabf669b993bbbc2f8"} Jul 21 15:01:42.171401 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:42.171351 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" podStartSLOduration=1.171336978 podStartE2EDuration="1.171336978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:01:42.168927875 +0000 UTC m=+2664.536437902" watchObservedRunningTime="2026-07-21 15:01:42.171336978 +0000 UTC m=+2664.538847056" Jul 21 15:01:46.766933 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:46.766875 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" podUID="e0ffe0c8-4224-4111-b0d2-d34e611aca09" containerName="main" containerID="cri-o://b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc" gracePeriod=45 Jul 21 15:01:47.023240 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.023177 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 15:01:47.035903 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.035880 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grdtq\" (UniqueName: \"kubernetes.io/projected/e0ffe0c8-4224-4111-b0d2-d34e611aca09-kube-api-access-grdtq\") pod \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " Jul 21 15:01:47.036047 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.035941 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tmp-dir\") pod \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " Jul 21 15:01:47.036047 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.035966 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tls-certs\") pod \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " Jul 21 15:01:47.036047 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.035985 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-model-cache\") pod \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " Jul 21 15:01:47.036047 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.036037 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-dshm\") pod \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " Jul 21 15:01:47.036263 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.036065 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-home\") pod \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\" (UID: \"e0ffe0c8-4224-4111-b0d2-d34e611aca09\") " Jul 21 15:01:47.036263 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.036195 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0ffe0c8-4224-4111-b0d2-d34e611aca09" (UID: "e0ffe0c8-4224-4111-b0d2-d34e611aca09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:01:47.036402 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.036354 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0ffe0c8-4224-4111-b0d2-d34e611aca09" (UID: "e0ffe0c8-4224-4111-b0d2-d34e611aca09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:01:47.036471 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.036398 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tmp-dir\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:01:47.036471 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.036405 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-home" (OuterVolumeSpecName: "home") pod "e0ffe0c8-4224-4111-b0d2-d34e611aca09" (UID: "e0ffe0c8-4224-4111-b0d2-d34e611aca09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:01:47.038228 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.038203 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-dshm" (OuterVolumeSpecName: "dshm") pod "e0ffe0c8-4224-4111-b0d2-d34e611aca09" (UID: "e0ffe0c8-4224-4111-b0d2-d34e611aca09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:01:47.038517 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.038486 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0ffe0c8-4224-4111-b0d2-d34e611aca09" (UID: "e0ffe0c8-4224-4111-b0d2-d34e611aca09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:01:47.038595 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.038509 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ffe0c8-4224-4111-b0d2-d34e611aca09-kube-api-access-grdtq" (OuterVolumeSpecName: "kube-api-access-grdtq") pod "e0ffe0c8-4224-4111-b0d2-d34e611aca09" (UID: "e0ffe0c8-4224-4111-b0d2-d34e611aca09"). InnerVolumeSpecName "kube-api-access-grdtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:01:47.137207 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.137174 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-dshm\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:01:47.137207 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.137204 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-home\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:01:47.137207 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.137213 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grdtq\" (UniqueName: \"kubernetes.io/projected/e0ffe0c8-4224-4111-b0d2-d34e611aca09-kube-api-access-grdtq\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:01:47.137433 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.137222 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0ffe0c8-4224-4111-b0d2-d34e611aca09-tls-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:01:47.137433 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.137231 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0ffe0c8-4224-4111-b0d2-d34e611aca09-model-cache\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:01:47.176560 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.176528 2570 generic.go:356] "Generic (PLEG): container finished" podID="e0ffe0c8-4224-4111-b0d2-d34e611aca09" containerID="b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc" exitCode=0 Jul 21 15:01:47.176744 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.176570 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" Jul 21 15:01:47.176744 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.176611 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" event={"ID":"e0ffe0c8-4224-4111-b0d2-d34e611aca09","Type":"ContainerDied","Data":"b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc"} Jul 21 15:01:47.176744 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.176646 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64" event={"ID":"e0ffe0c8-4224-4111-b0d2-d34e611aca09","Type":"ContainerDied","Data":"d3f78d8aac451ae389b549881816c190818ac34118520e47d2e746c52038b9b6"} Jul 21 15:01:47.176744 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.176662 2570 scope.go:117] "RemoveContainer" containerID="b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc" Jul 21 15:01:47.189239 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.189168 2570 scope.go:117] "RemoveContainer" containerID="b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc" Jul 21 15:01:47.189532 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:01:47.189502 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc\": container with ID starting with b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc not found: ID does not exist" containerID="b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc" Jul 21 15:01:47.189649 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.189541 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc"} err="failed to get container status \"b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc\": rpc error: code = NotFound desc = could not find container \"b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc\": container with ID starting with b8bcf4fcd14f88a23e6425abb093ff7d854db9f567cac2c27e88c80c0f863bcc not found: ID does not exist" Jul 21 15:01:47.207077 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.207051 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64"] Jul 21 15:01:47.210238 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:47.210217 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/auth-test-model-d89949-kserve-65ccbf67f9-r7v64"] Jul 21 15:01:48.141152 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:48.141129 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0ffe0c8-4224-4111-b0d2-d34e611aca09" path="/var/lib/kubelet/pods/e0ffe0c8-4224-4111-b0d2-d34e611aca09/volumes" Jul 21 15:01:51.794689 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:51.794655 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:51.795090 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:51.794702 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:51.807124 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:51.807100 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:01:52.213961 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:01:52.213936 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:02:07.990821 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:07.990774 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:18.310353 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:18.310272 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 15:02:18.318387 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:18.318364 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 15:02:19.582157 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.582122 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5"] Jul 21 15:02:19.582624 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.582575 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ffe0c8-4224-4111-b0d2-d34e611aca09" containerName="main" Jul 21 15:02:19.582624 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.582588 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ffe0c8-4224-4111-b0d2-d34e611aca09" containerName="main" Jul 21 15:02:19.582700 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.582671 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ffe0c8-4224-4111-b0d2-d34e611aca09" containerName="main" Jul 21 15:02:19.587182 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.587164 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.591596 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.591572 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"kube-root-ca.crt\"" Jul 21 15:02:19.591722 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.591618 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"sub-test-model-449411-kserve-self-signed-certs\"" Jul 21 15:02:19.591722 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.591572 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"default-dockercfg-jw7fr\"" Jul 21 15:02:19.591722 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.591619 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"openshift-service-ca.crt\"" Jul 21 15:02:19.597237 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.597212 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5"] Jul 21 15:02:19.665314 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.665286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:19.698773 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.698748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-home\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.698889 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.698777 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-tmp-dir\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.698889 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.698809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-dshm\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.698889 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.698848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvkg\" (UniqueName: \"kubernetes.io/projected/ebe1147a-ad98-4ffc-9508-0838028a1d05-kube-api-access-plvkg\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.699030 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.698895 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-model-cache\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.699030 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.698966 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe1147a-ad98-4ffc-9508-0838028a1d05-tls-certs\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.799787 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.799761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-model-cache\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.799916 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.799800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe1147a-ad98-4ffc-9508-0838028a1d05-tls-certs\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.799916 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.799828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-home\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.800072 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.799930 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-tmp-dir\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.800072 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.799998 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-dshm\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.800072 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.800035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvkg\" (UniqueName: \"kubernetes.io/projected/ebe1147a-ad98-4ffc-9508-0838028a1d05-kube-api-access-plvkg\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.800293 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.800266 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-model-cache\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.800293 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.800287 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-home\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.800452 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.800317 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-tmp-dir\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.801986 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.801967 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-dshm\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.802319 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.802301 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe1147a-ad98-4ffc-9508-0838028a1d05-tls-certs\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.810287 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.810265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvkg\" (UniqueName: \"kubernetes.io/projected/ebe1147a-ad98-4ffc-9508-0838028a1d05-kube-api-access-plvkg\") pod \"sub-test-model-449411-kserve-68bf556b9-lwhm5\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:19.897608 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:19.897586 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:20.042361 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:20.042332 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5"] Jul 21 15:02:20.047151 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:02:20.047125 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebe1147a_ad98_4ffc_9508_0838028a1d05.slice/crio-57fcf370754ee7fbe30982d2f8c08f9fe5fe7209e59d585ee0760b3526e44fd9 WatchSource:0}: Error finding container 57fcf370754ee7fbe30982d2f8c08f9fe5fe7209e59d585ee0760b3526e44fd9: Status 404 returned error can't find the container with id 57fcf370754ee7fbe30982d2f8c08f9fe5fe7209e59d585ee0760b3526e44fd9 Jul 21 15:02:20.352235 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:20.352155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" event={"ID":"ebe1147a-ad98-4ffc-9508-0838028a1d05","Type":"ContainerStarted","Data":"d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7"} Jul 21 15:02:20.352235 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:20.352193 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" event={"ID":"ebe1147a-ad98-4ffc-9508-0838028a1d05","Type":"ContainerStarted","Data":"57fcf370754ee7fbe30982d2f8c08f9fe5fe7209e59d585ee0760b3526e44fd9"} Jul 21 15:02:20.377916 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:20.377867 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" podStartSLOduration=1.377853688 podStartE2EDuration="1.377853688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:02:20.376790929 +0000 UTC m=+2702.744300956" watchObservedRunningTime="2026-07-21 15:02:20.377853688 +0000 UTC m=+2702.745363715" Jul 21 15:02:29.897848 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:29.897808 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:29.898245 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:29.897864 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:29.909902 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:29.909880 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:30.414115 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:30.414087 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:02:38.225869 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:38.225830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:02:38.951812 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:02:38.951772 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:03.466795 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:03.466757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:04.972211 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:04.972167 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g"] Jul 21 15:03:04.994203 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:04.994169 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:06.173421 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:06.173389 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5"] Jul 21 15:03:06.188836 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:06.188808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:15.946830 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:15.946794 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:18.485691 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:18.485662 2570 scope.go:117] "RemoveContainer" containerID="a7a3e8b0035ffee022dcf54fae82b90a466c5d2c7f9e6c3c83548eb3d92ed7cd" Jul 21 15:03:18.494174 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:18.494155 2570 scope.go:117] "RemoveContainer" containerID="405e51573f5c3145cf4739ceb76c911fa21fc0e3ada5238ebd781375709217f9" Jul 21 15:03:19.991499 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:19.991460 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" podUID="fcedfb22-805c-4a1c-a376-4244be61a94d" containerName="main" containerID="cri-o://ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c" gracePeriod=45 Jul 21 15:03:20.233457 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.233434 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:03:20.265221 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265165 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcedfb22-805c-4a1c-a376-4244be61a94d-tls-certs\") pod \"fcedfb22-805c-4a1c-a376-4244be61a94d\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " Jul 21 15:03:20.265349 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265224 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-model-cache\") pod \"fcedfb22-805c-4a1c-a376-4244be61a94d\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " Jul 21 15:03:20.265349 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265258 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-tmp-dir\") pod \"fcedfb22-805c-4a1c-a376-4244be61a94d\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " Jul 21 15:03:20.265467 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265347 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mppg4\" (UniqueName: \"kubernetes.io/projected/fcedfb22-805c-4a1c-a376-4244be61a94d-kube-api-access-mppg4\") pod \"fcedfb22-805c-4a1c-a376-4244be61a94d\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " Jul 21 15:03:20.265467 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265412 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-dshm\") pod \"fcedfb22-805c-4a1c-a376-4244be61a94d\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " Jul 21 15:03:20.265467 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265434 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-home\") pod \"fcedfb22-805c-4a1c-a376-4244be61a94d\" (UID: \"fcedfb22-805c-4a1c-a376-4244be61a94d\") " Jul 21 15:03:20.265630 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265535 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcedfb22-805c-4a1c-a376-4244be61a94d" (UID: "fcedfb22-805c-4a1c-a376-4244be61a94d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:20.265763 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265743 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-home" (OuterVolumeSpecName: "home") pod "fcedfb22-805c-4a1c-a376-4244be61a94d" (UID: "fcedfb22-805c-4a1c-a376-4244be61a94d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:20.265851 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265766 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-tmp-dir\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:20.265851 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.265823 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcedfb22-805c-4a1c-a376-4244be61a94d" (UID: "fcedfb22-805c-4a1c-a376-4244be61a94d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:20.267256 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.267233 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcedfb22-805c-4a1c-a376-4244be61a94d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcedfb22-805c-4a1c-a376-4244be61a94d" (UID: "fcedfb22-805c-4a1c-a376-4244be61a94d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:03:20.267402 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.267377 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcedfb22-805c-4a1c-a376-4244be61a94d-kube-api-access-mppg4" (OuterVolumeSpecName: "kube-api-access-mppg4") pod "fcedfb22-805c-4a1c-a376-4244be61a94d" (UID: "fcedfb22-805c-4a1c-a376-4244be61a94d"). InnerVolumeSpecName "kube-api-access-mppg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:03:20.267491 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.267420 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-dshm" (OuterVolumeSpecName: "dshm") pod "fcedfb22-805c-4a1c-a376-4244be61a94d" (UID: "fcedfb22-805c-4a1c-a376-4244be61a94d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:20.366481 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.366453 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mppg4\" (UniqueName: \"kubernetes.io/projected/fcedfb22-805c-4a1c-a376-4244be61a94d-kube-api-access-mppg4\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:20.366481 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.366476 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-dshm\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:20.366481 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.366485 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-home\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:20.366644 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.366492 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcedfb22-805c-4a1c-a376-4244be61a94d-tls-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:20.366644 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.366502 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcedfb22-805c-4a1c-a376-4244be61a94d-model-cache\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:20.676766 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.676733 2570 generic.go:356] "Generic (PLEG): container finished" podID="fcedfb22-805c-4a1c-a376-4244be61a94d" containerID="ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c" exitCode=0 Jul 21 15:03:20.676885 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.676771 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" Jul 21 15:03:20.676885 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.676792 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" event={"ID":"fcedfb22-805c-4a1c-a376-4244be61a94d","Type":"ContainerDied","Data":"ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c"} Jul 21 15:03:20.676885 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.676836 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g" event={"ID":"fcedfb22-805c-4a1c-a376-4244be61a94d","Type":"ContainerDied","Data":"bb63bebb33366dc559c765e0d7572ace2d055dae5af1abbabf669b993bbbc2f8"} Jul 21 15:03:20.676885 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.676855 2570 scope.go:117] "RemoveContainer" containerID="ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c" Jul 21 15:03:20.685745 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.685724 2570 scope.go:117] "RemoveContainer" containerID="ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c" Jul 21 15:03:20.686000 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:03:20.685982 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c\": container with ID starting with ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c not found: ID does not exist" containerID="ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c" Jul 21 15:03:20.686077 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.686037 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c"} err="failed to get container status \"ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c\": rpc error: code = NotFound desc = could not find container \"ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c\": container with ID starting with ff9809dd89bbb24b1ad4c91b218a5c51683c8ad0a81e0da6820607d6406e7d5c not found: ID does not exist" Jul 21 15:03:20.706083 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.706059 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g"] Jul 21 15:03:20.711902 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:20.711880 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d89949/sub-test-model-d89949-kserve-5c49b977cc-txg7g"] Jul 21 15:03:21.192850 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.192818 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" podUID="ebe1147a-ad98-4ffc-9508-0838028a1d05" containerName="main" containerID="cri-o://d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7" gracePeriod=45 Jul 21 15:03:21.439979 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.439951 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:03:21.474689 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.474620 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-home\") pod \"ebe1147a-ad98-4ffc-9508-0838028a1d05\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " Jul 21 15:03:21.474689 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.474661 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plvkg\" (UniqueName: \"kubernetes.io/projected/ebe1147a-ad98-4ffc-9508-0838028a1d05-kube-api-access-plvkg\") pod \"ebe1147a-ad98-4ffc-9508-0838028a1d05\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " Jul 21 15:03:21.474875 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.474712 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe1147a-ad98-4ffc-9508-0838028a1d05-tls-certs\") pod \"ebe1147a-ad98-4ffc-9508-0838028a1d05\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " Jul 21 15:03:21.474875 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.474779 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-tmp-dir\") pod \"ebe1147a-ad98-4ffc-9508-0838028a1d05\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " Jul 21 15:03:21.475055 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.474969 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-home" (OuterVolumeSpecName: "home") pod "ebe1147a-ad98-4ffc-9508-0838028a1d05" (UID: "ebe1147a-ad98-4ffc-9508-0838028a1d05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:21.475128 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.475080 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebe1147a-ad98-4ffc-9508-0838028a1d05" (UID: "ebe1147a-ad98-4ffc-9508-0838028a1d05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:21.475181 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.475127 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-dshm\") pod \"ebe1147a-ad98-4ffc-9508-0838028a1d05\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " Jul 21 15:03:21.475235 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.475198 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-model-cache\") pod \"ebe1147a-ad98-4ffc-9508-0838028a1d05\" (UID: \"ebe1147a-ad98-4ffc-9508-0838028a1d05\") " Jul 21 15:03:21.475497 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.475469 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-home\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:21.475497 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.475493 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-tmp-dir\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:21.475824 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.475783 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebe1147a-ad98-4ffc-9508-0838028a1d05" (UID: "ebe1147a-ad98-4ffc-9508-0838028a1d05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:21.477551 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.477529 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe1147a-ad98-4ffc-9508-0838028a1d05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebe1147a-ad98-4ffc-9508-0838028a1d05" (UID: "ebe1147a-ad98-4ffc-9508-0838028a1d05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:03:21.477551 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.477541 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe1147a-ad98-4ffc-9508-0838028a1d05-kube-api-access-plvkg" (OuterVolumeSpecName: "kube-api-access-plvkg") pod "ebe1147a-ad98-4ffc-9508-0838028a1d05" (UID: "ebe1147a-ad98-4ffc-9508-0838028a1d05"). InnerVolumeSpecName "kube-api-access-plvkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:03:21.477673 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.477527 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-dshm" (OuterVolumeSpecName: "dshm") pod "ebe1147a-ad98-4ffc-9508-0838028a1d05" (UID: "ebe1147a-ad98-4ffc-9508-0838028a1d05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:03:21.576422 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.576383 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe1147a-ad98-4ffc-9508-0838028a1d05-tls-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:21.576422 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.576417 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-dshm\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:21.576422 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.576426 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebe1147a-ad98-4ffc-9508-0838028a1d05-model-cache\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:21.576599 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.576435 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvkg\" (UniqueName: \"kubernetes.io/projected/ebe1147a-ad98-4ffc-9508-0838028a1d05-kube-api-access-plvkg\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:03:21.684097 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.684069 2570 generic.go:356] "Generic (PLEG): container finished" podID="ebe1147a-ad98-4ffc-9508-0838028a1d05" containerID="d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7" exitCode=0 Jul 21 15:03:21.684217 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.684108 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" Jul 21 15:03:21.684217 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.684133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" event={"ID":"ebe1147a-ad98-4ffc-9508-0838028a1d05","Type":"ContainerDied","Data":"d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7"} Jul 21 15:03:21.684217 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.684172 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5" event={"ID":"ebe1147a-ad98-4ffc-9508-0838028a1d05","Type":"ContainerDied","Data":"57fcf370754ee7fbe30982d2f8c08f9fe5fe7209e59d585ee0760b3526e44fd9"} Jul 21 15:03:21.684217 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.684194 2570 scope.go:117] "RemoveContainer" containerID="d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7" Jul 21 15:03:21.693874 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.693860 2570 scope.go:117] "RemoveContainer" containerID="d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7" Jul 21 15:03:21.694154 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:03:21.694133 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7\": container with ID starting with d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7 not found: ID does not exist" containerID="d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7" Jul 21 15:03:21.694207 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.694162 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7"} err="failed to get container status \"d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7\": rpc error: code = NotFound desc = could not find container \"d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7\": container with ID starting with d3cd43c06b59f9f0ee6fcfba635858b41a1dbd1139da35f22755ed1d9ddf64c7 not found: ID does not exist" Jul 21 15:03:21.724123 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.721714 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5"] Jul 21 15:03:21.726475 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:21.726425 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-449411/sub-test-model-449411-kserve-68bf556b9-lwhm5"] Jul 21 15:03:22.143849 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:22.143819 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe1147a-ad98-4ffc-9508-0838028a1d05" path="/var/lib/kubelet/pods/ebe1147a-ad98-4ffc-9508-0838028a1d05/volumes" Jul 21 15:03:22.144224 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:22.144210 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcedfb22-805c-4a1c-a376-4244be61a94d" path="/var/lib/kubelet/pods/fcedfb22-805c-4a1c-a376-4244be61a94d/volumes" Jul 21 15:03:35.183212 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:35.183177 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:35.284581 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:35.284546 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:44.471088 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.470999 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k"] Jul 21 15:03:44.471452 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.471386 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe1147a-ad98-4ffc-9508-0838028a1d05" containerName="main" Jul 21 15:03:44.471452 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.471397 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe1147a-ad98-4ffc-9508-0838028a1d05" containerName="main" Jul 21 15:03:44.471452 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.471413 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcedfb22-805c-4a1c-a376-4244be61a94d" containerName="main" Jul 21 15:03:44.471452 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.471418 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcedfb22-805c-4a1c-a376-4244be61a94d" containerName="main" Jul 21 15:03:44.471617 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.471486 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe1147a-ad98-4ffc-9508-0838028a1d05" containerName="main" Jul 21 15:03:44.471617 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.471498 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcedfb22-805c-4a1c-a376-4244be61a94d" containerName="main" Jul 21 15:03:44.474338 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.474321 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.477994 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.477966 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"default-dockercfg-jw7fr\"" Jul 21 15:03:44.478142 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.478038 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"kube-root-ca.crt\"" Jul 21 15:03:44.478142 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.478076 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"openshift-service-ca.crt\"" Jul 21 15:03:44.478142 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.478038 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-449411\"/\"rate-test-model-449411-kserve-self-signed-certs\"" Jul 21 15:03:44.484512 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.484487 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k"] Jul 21 15:03:44.522262 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.522213 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:03:44.553932 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.553900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-tmp-dir\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.554112 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.553960 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-model-cache\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.554112 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.553987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlwqk\" (UniqueName: \"kubernetes.io/projected/b63c2838-d5a9-40e7-a865-1b45061a93ba-kube-api-access-tlwqk\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.554112 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.554038 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-home\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.554240 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.554137 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63c2838-d5a9-40e7-a865-1b45061a93ba-tls-certs\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.554240 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.554195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-dshm\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.654956 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.654916 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-dshm\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655198 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.654983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-tmp-dir\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655198 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-model-cache\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655198 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlwqk\" (UniqueName: \"kubernetes.io/projected/b63c2838-d5a9-40e7-a865-1b45061a93ba-kube-api-access-tlwqk\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655198 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-home\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655198 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655161 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63c2838-d5a9-40e7-a865-1b45061a93ba-tls-certs\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655496 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655455 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-tmp-dir\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655577 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655496 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-model-cache\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.655577 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.655529 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-home\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.657296 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.657274 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-dshm\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.657561 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.657542 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63c2838-d5a9-40e7-a865-1b45061a93ba-tls-certs\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.665396 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.665373 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlwqk\" (UniqueName: \"kubernetes.io/projected/b63c2838-d5a9-40e7-a865-1b45061a93ba-kube-api-access-tlwqk\") pod \"rate-test-model-449411-kserve-584df79dff-w5n5k\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.784547 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.784464 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:44.919506 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:44.919483 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k"] Jul 21 15:03:44.921607 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:03:44.921576 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb63c2838_d5a9_40e7_a865_1b45061a93ba.slice/crio-1189a9d026ae66bab6a0ea2d490cd3e822cfc6494aa09e5d9a29fbae846616ca WatchSource:0}: Error finding container 1189a9d026ae66bab6a0ea2d490cd3e822cfc6494aa09e5d9a29fbae846616ca: Status 404 returned error can't find the container with id 1189a9d026ae66bab6a0ea2d490cd3e822cfc6494aa09e5d9a29fbae846616ca Jul 21 15:03:45.813251 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:45.813215 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" event={"ID":"b63c2838-d5a9-40e7-a865-1b45061a93ba","Type":"ContainerStarted","Data":"a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993"} Jul 21 15:03:45.813615 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:45.813262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" event={"ID":"b63c2838-d5a9-40e7-a865-1b45061a93ba","Type":"ContainerStarted","Data":"1189a9d026ae66bab6a0ea2d490cd3e822cfc6494aa09e5d9a29fbae846616ca"} Jul 21 15:03:45.837861 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:45.837813 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" podStartSLOduration=1.837798656 podStartE2EDuration="1.837798656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 15:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:03:45.835714346 +0000 UTC m=+2788.203224374" watchObservedRunningTime="2026-07-21 15:03:45.837798656 +0000 UTC m=+2788.205308683" Jul 21 15:03:54.785244 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:54.785208 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:54.785728 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:54.785261 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:54.797105 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:54.797064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:03:54.868635 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:03:54.868611 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:04:06.056113 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:06.056063 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:06.132150 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:06.132109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:14.662336 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:14.662303 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:14.762567 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:14.762527 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:18.517082 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:18.517049 2570 scope.go:117] "RemoveContainer" containerID="5027e040aaf22121d4a6ae5c5283277ad1d7579658956ad30e3f5b2ac2e4f2a4" Jul 21 15:04:18.525559 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:18.525540 2570 scope.go:117] "RemoveContainer" containerID="1d5ba565c0882fc3e31b3d1e406ae59dcf84db6d2b8f087e76308448a2590fdc" Jul 21 15:04:27.884055 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:27.883997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:28.807364 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:28.807328 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k"] Jul 21 15:04:29.352224 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.352182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:29.814200 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.814161 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-140533221018288"] Jul 21 15:04:29.817718 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.817701 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:29.820580 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.820557 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-g6xtp\"" Jul 21 15:04:29.825715 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.825691 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533221018288"] Jul 21 15:04:29.859584 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.859550 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rg88\" (UniqueName: \"kubernetes.io/projected/47fcdc92-8e69-4b93-be41-954006958742-kube-api-access-4rg88\") pod \"test-curl-7696-140533221018288\" (UID: \"47fcdc92-8e69-4b93-be41-954006958742\") " pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:29.960229 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.960190 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rg88\" (UniqueName: \"kubernetes.io/projected/47fcdc92-8e69-4b93-be41-954006958742-kube-api-access-4rg88\") pod \"test-curl-7696-140533221018288\" (UID: \"47fcdc92-8e69-4b93-be41-954006958742\") " pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:29.970934 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:29.970902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rg88\" (UniqueName: \"kubernetes.io/projected/47fcdc92-8e69-4b93-be41-954006958742-kube-api-access-4rg88\") pod \"test-curl-7696-140533221018288\" (UID: \"47fcdc92-8e69-4b93-be41-954006958742\") " pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:30.128279 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:30.128166 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:30.266818 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:30.266788 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533221018288"] Jul 21 15:04:30.269034 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:30.268994 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47fcdc92_8e69_4b93_be41_954006958742.slice/crio-6454fda02e4e3179e03112b6e91b2e8b9b512d13a00b6aecce3efb7128e56b5a WatchSource:0}: Error finding container 6454fda02e4e3179e03112b6e91b2e8b9b512d13a00b6aecce3efb7128e56b5a: Status 404 returned error can't find the container with id 6454fda02e4e3179e03112b6e91b2e8b9b512d13a00b6aecce3efb7128e56b5a Jul 21 15:04:30.270721 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:30.270704 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 15:04:31.051093 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:31.051064 2570 generic.go:356] "Generic (PLEG): container finished" podID="47fcdc92-8e69-4b93-be41-954006958742" containerID="5d4ace68e9b1d8be79ae94d01d58bb1f4ec9b34599b4d14dd49ac928ed7033a2" exitCode=0 Jul 21 15:04:31.051450 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:31.051142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533221018288" event={"ID":"47fcdc92-8e69-4b93-be41-954006958742","Type":"ContainerDied","Data":"5d4ace68e9b1d8be79ae94d01d58bb1f4ec9b34599b4d14dd49ac928ed7033a2"} Jul 21 15:04:31.051450 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:31.051180 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533221018288" event={"ID":"47fcdc92-8e69-4b93-be41-954006958742","Type":"ContainerStarted","Data":"6454fda02e4e3179e03112b6e91b2e8b9b512d13a00b6aecce3efb7128e56b5a"} Jul 21 15:04:32.188285 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.188258 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:32.233980 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.233948 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-140533221018288_47fcdc92-8e69-4b93-be41-954006958742/test-curl-7696-140533221018288/0.log" Jul 21 15:04:32.266597 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.266562 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-140533221018288"] Jul 21 15:04:32.275002 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.274971 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-140533221018288"] Jul 21 15:04:32.279128 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.279109 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rg88\" (UniqueName: \"kubernetes.io/projected/47fcdc92-8e69-4b93-be41-954006958742-kube-api-access-4rg88\") pod \"47fcdc92-8e69-4b93-be41-954006958742\" (UID: \"47fcdc92-8e69-4b93-be41-954006958742\") " Jul 21 15:04:32.281163 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.281143 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47fcdc92-8e69-4b93-be41-954006958742-kube-api-access-4rg88" (OuterVolumeSpecName: "kube-api-access-4rg88") pod "47fcdc92-8e69-4b93-be41-954006958742" (UID: "47fcdc92-8e69-4b93-be41-954006958742"). InnerVolumeSpecName "kube-api-access-4rg88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:32.380068 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.379967 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rg88\" (UniqueName: \"kubernetes.io/projected/47fcdc92-8e69-4b93-be41-954006958742-kube-api-access-4rg88\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:32.411779 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.411745 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-140533221107760"] Jul 21 15:04:32.412181 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.412167 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fcdc92-8e69-4b93-be41-954006958742" containerName="test-curl-7696-140533221018288" Jul 21 15:04:32.412229 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.412182 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fcdc92-8e69-4b93-be41-954006958742" containerName="test-curl-7696-140533221018288" Jul 21 15:04:32.412264 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.412252 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fcdc92-8e69-4b93-be41-954006958742" containerName="test-curl-7696-140533221018288" Jul 21 15:04:32.415444 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.415430 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:32.427536 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.427503 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533221107760"] Jul 21 15:04:32.480530 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.480491 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn4x\" (UniqueName: \"kubernetes.io/projected/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1-kube-api-access-lxn4x\") pod \"test-curl-7696-140533221107760\" (UID: \"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1\") " pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:32.581363 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.581325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn4x\" (UniqueName: \"kubernetes.io/projected/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1-kube-api-access-lxn4x\") pod \"test-curl-7696-140533221107760\" (UID: \"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1\") " pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:32.596096 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.596065 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn4x\" (UniqueName: \"kubernetes.io/projected/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1-kube-api-access-lxn4x\") pod \"test-curl-7696-140533221107760\" (UID: \"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1\") " pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:32.724212 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.724176 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:32.883409 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:32.883371 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72b436f7_2b8e_45c5_b669_69d9f3f2c9c1.slice/crio-fa2978a490f765dae8f2e21d1ef05a1cbedded307551315c53e3eab409d94593 WatchSource:0}: Error finding container fa2978a490f765dae8f2e21d1ef05a1cbedded307551315c53e3eab409d94593: Status 404 returned error can't find the container with id fa2978a490f765dae8f2e21d1ef05a1cbedded307551315c53e3eab409d94593 Jul 21 15:04:32.883714 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:32.883693 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533221107760"] Jul 21 15:04:33.063053 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:33.062950 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:33.063053 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:33.063001 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6454fda02e4e3179e03112b6e91b2e8b9b512d13a00b6aecce3efb7128e56b5a" Jul 21 15:04:33.064370 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:33.064334 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533221107760" event={"ID":"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1","Type":"ContainerStarted","Data":"fa2978a490f765dae8f2e21d1ef05a1cbedded307551315c53e3eab409d94593"} Jul 21 15:04:33.078426 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:04:33.078398 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-140533221018288\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="47fcdc92-8e69-4b93-be41-954006958742" pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:34.070235 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:34.070202 2570 generic.go:356] "Generic (PLEG): container finished" podID="72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" containerID="db8295dbeec13c0226fbb81fbf978552860d98fdb1c5ea1dcc36803c1765365a" exitCode=0 Jul 21 15:04:34.070636 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:34.070242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533221107760" event={"ID":"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1","Type":"ContainerDied","Data":"db8295dbeec13c0226fbb81fbf978552860d98fdb1c5ea1dcc36803c1765365a"} Jul 21 15:04:34.078906 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:04:34.078880 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-140533221018288\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="47fcdc92-8e69-4b93-be41-954006958742" pod="opendatahub/test-curl-7696-140533221018288" Jul 21 15:04:34.141362 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:34.141327 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fcdc92-8e69-4b93-be41-954006958742" path="/var/lib/kubelet/pods/47fcdc92-8e69-4b93-be41-954006958742/volumes" Jul 21 15:04:35.213549 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.213523 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:35.252035 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.251989 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-140533221107760_72b436f7-2b8e-45c5-b669-69d9f3f2c9c1/test-curl-7696-140533221107760/0.log" Jul 21 15:04:35.291176 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.291145 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-140533221107760"] Jul 21 15:04:35.295125 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.295096 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-140533221107760"] Jul 21 15:04:35.305310 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.305289 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxn4x\" (UniqueName: \"kubernetes.io/projected/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1-kube-api-access-lxn4x\") pod \"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1\" (UID: \"72b436f7-2b8e-45c5-b669-69d9f3f2c9c1\") " Jul 21 15:04:35.307321 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.307301 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1-kube-api-access-lxn4x" (OuterVolumeSpecName: "kube-api-access-lxn4x") pod "72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" (UID: "72b436f7-2b8e-45c5-b669-69d9f3f2c9c1"). InnerVolumeSpecName "kube-api-access-lxn4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:35.406730 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.406698 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxn4x\" (UniqueName: \"kubernetes.io/projected/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1-kube-api-access-lxn4x\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:35.468619 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.468584 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:35.468989 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.468976 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" containerName="test-curl-7696-140533221107760" Jul 21 15:04:35.468989 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.468990 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" containerName="test-curl-7696-140533221107760" Jul 21 15:04:35.469093 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.469082 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" containerName="test-curl-7696-140533221107760" Jul 21 15:04:35.472093 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.472075 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:35.483670 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.483645 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:35.507901 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.507869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjcd\" (UniqueName: \"kubernetes.io/projected/391248fe-8059-4d1f-aee5-aa633b9e96e2-kube-api-access-pdjcd\") pod \"test-curl-7696-140533220988080\" (UID: \"391248fe-8059-4d1f-aee5-aa633b9e96e2\") " pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:35.608885 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.608850 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjcd\" (UniqueName: \"kubernetes.io/projected/391248fe-8059-4d1f-aee5-aa633b9e96e2-kube-api-access-pdjcd\") pod \"test-curl-7696-140533220988080\" (UID: \"391248fe-8059-4d1f-aee5-aa633b9e96e2\") " pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:35.623975 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.623948 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjcd\" (UniqueName: \"kubernetes.io/projected/391248fe-8059-4d1f-aee5-aa633b9e96e2-kube-api-access-pdjcd\") pod \"test-curl-7696-140533220988080\" (UID: \"391248fe-8059-4d1f-aee5-aa633b9e96e2\") " pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:35.781214 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.781128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:35.933768 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:35.933744 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:35.935244 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:35.935219 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod391248fe_8059_4d1f_aee5_aa633b9e96e2.slice/crio-8fa8b0b4181253f4e859691dfdf46e5fb78004aebcc5c4c200878490a7b30539 WatchSource:0}: Error finding container 8fa8b0b4181253f4e859691dfdf46e5fb78004aebcc5c4c200878490a7b30539: Status 404 returned error can't find the container with id 8fa8b0b4181253f4e859691dfdf46e5fb78004aebcc5c4c200878490a7b30539 Jul 21 15:04:36.080905 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:36.080828 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:36.080905 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:36.080845 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2978a490f765dae8f2e21d1ef05a1cbedded307551315c53e3eab409d94593" Jul 21 15:04:36.081890 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:36.081859 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220988080" event={"ID":"391248fe-8059-4d1f-aee5-aa633b9e96e2","Type":"ContainerStarted","Data":"8fa8b0b4181253f4e859691dfdf46e5fb78004aebcc5c4c200878490a7b30539"} Jul 21 15:04:36.092041 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:04:36.092001 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-140533221107760\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" pod="opendatahub/test-curl-7696-140533221107760" Jul 21 15:04:36.141758 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:36.141725 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b436f7-2b8e-45c5-b669-69d9f3f2c9c1" path="/var/lib/kubelet/pods/72b436f7-2b8e-45c5-b669-69d9f3f2c9c1/volumes" Jul 21 15:04:37.088126 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:37.088092 2570 generic.go:356] "Generic (PLEG): container finished" podID="391248fe-8059-4d1f-aee5-aa633b9e96e2" containerID="346444ab78e4740ae5c5001f10e060d393ed4c28117943ba3b6487f6785fe227" exitCode=0 Jul 21 15:04:37.088514 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:37.088150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220988080" event={"ID":"391248fe-8059-4d1f-aee5-aa633b9e96e2","Type":"ContainerDied","Data":"346444ab78e4740ae5c5001f10e060d393ed4c28117943ba3b6487f6785fe227"} Jul 21 15:04:38.230172 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.230145 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:38.260965 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.260931 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-140533220988080_391248fe-8059-4d1f-aee5-aa633b9e96e2/test-curl-7696-140533220988080/0.log" Jul 21 15:04:38.294236 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.294202 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:38.300366 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.300341 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:38.332328 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.332285 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdjcd\" (UniqueName: \"kubernetes.io/projected/391248fe-8059-4d1f-aee5-aa633b9e96e2-kube-api-access-pdjcd\") pod \"391248fe-8059-4d1f-aee5-aa633b9e96e2\" (UID: \"391248fe-8059-4d1f-aee5-aa633b9e96e2\") " Jul 21 15:04:38.334393 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.334366 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391248fe-8059-4d1f-aee5-aa633b9e96e2-kube-api-access-pdjcd" (OuterVolumeSpecName: "kube-api-access-pdjcd") pod "391248fe-8059-4d1f-aee5-aa633b9e96e2" (UID: "391248fe-8059-4d1f-aee5-aa633b9e96e2"). InnerVolumeSpecName "kube-api-access-pdjcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:38.425066 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.425028 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:38.425540 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.425521 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391248fe-8059-4d1f-aee5-aa633b9e96e2" containerName="test-curl-7696-140533220988080" Jul 21 15:04:38.425623 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.425542 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="391248fe-8059-4d1f-aee5-aa633b9e96e2" containerName="test-curl-7696-140533220988080" Jul 21 15:04:38.425671 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.425633 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="391248fe-8059-4d1f-aee5-aa633b9e96e2" containerName="test-curl-7696-140533220988080" Jul 21 15:04:38.432874 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.429522 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:38.433529 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.433508 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjcd\" (UniqueName: \"kubernetes.io/projected/391248fe-8059-4d1f-aee5-aa633b9e96e2-kube-api-access-pdjcd\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:38.438826 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.438769 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:38.534370 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.534330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24rt\" (UniqueName: \"kubernetes.io/projected/18658936-3202-40fc-b4db-a9f7f643d17b-kube-api-access-p24rt\") pod \"test-curl-7696-140533220988080\" (UID: \"18658936-3202-40fc-b4db-a9f7f643d17b\") " pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:38.635171 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.635127 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24rt\" (UniqueName: \"kubernetes.io/projected/18658936-3202-40fc-b4db-a9f7f643d17b-kube-api-access-p24rt\") pod \"test-curl-7696-140533220988080\" (UID: \"18658936-3202-40fc-b4db-a9f7f643d17b\") " pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:38.647840 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.647810 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24rt\" (UniqueName: \"kubernetes.io/projected/18658936-3202-40fc-b4db-a9f7f643d17b-kube-api-access-p24rt\") pod \"test-curl-7696-140533220988080\" (UID: \"18658936-3202-40fc-b4db-a9f7f643d17b\") " pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:38.743115 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.742983 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:38.883039 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:38.882987 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:38.886240 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:38.886212 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18658936_3202_40fc_b4db_a9f7f643d17b.slice/crio-61743f72012e8445c7da985a765fa70509b1067faca59f6087a5f5f628f6be8c WatchSource:0}: Error finding container 61743f72012e8445c7da985a765fa70509b1067faca59f6087a5f5f628f6be8c: Status 404 returned error can't find the container with id 61743f72012e8445c7da985a765fa70509b1067faca59f6087a5f5f628f6be8c Jul 21 15:04:39.099526 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:39.099439 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:39.099526 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:39.099467 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa8b0b4181253f4e859691dfdf46e5fb78004aebcc5c4c200878490a7b30539" Jul 21 15:04:39.100702 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:39.100674 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220988080" event={"ID":"18658936-3202-40fc-b4db-a9f7f643d17b","Type":"ContainerStarted","Data":"61743f72012e8445c7da985a765fa70509b1067faca59f6087a5f5f628f6be8c"} Jul 21 15:04:39.112267 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:39.112237 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7696-140533220988080" oldPodUID="391248fe-8059-4d1f-aee5-aa633b9e96e2" podUID="18658936-3202-40fc-b4db-a9f7f643d17b" Jul 21 15:04:40.108206 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:40.108167 2570 generic.go:356] "Generic (PLEG): container finished" podID="18658936-3202-40fc-b4db-a9f7f643d17b" containerID="3b78cad447c8942f6b3c7ed75f194c40e776e7ef1062607ce6437590345b9fb3" exitCode=0 Jul 21 15:04:40.108629 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:40.108251 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220988080" event={"ID":"18658936-3202-40fc-b4db-a9f7f643d17b","Type":"ContainerDied","Data":"3b78cad447c8942f6b3c7ed75f194c40e776e7ef1062607ce6437590345b9fb3"} Jul 21 15:04:40.141252 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:40.141223 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="391248fe-8059-4d1f-aee5-aa633b9e96e2" path="/var/lib/kubelet/pods/391248fe-8059-4d1f-aee5-aa633b9e96e2/volumes" Jul 21 15:04:41.246952 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.246929 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:41.277115 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.277067 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-140533220988080_18658936-3202-40fc-b4db-a9f7f643d17b/test-curl-7696-140533220988080/0.log" Jul 21 15:04:41.307840 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.307807 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:41.311882 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.311858 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-140533220988080"] Jul 21 15:04:41.361285 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.361252 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p24rt\" (UniqueName: \"kubernetes.io/projected/18658936-3202-40fc-b4db-a9f7f643d17b-kube-api-access-p24rt\") pod \"18658936-3202-40fc-b4db-a9f7f643d17b\" (UID: \"18658936-3202-40fc-b4db-a9f7f643d17b\") " Jul 21 15:04:41.363353 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.363321 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18658936-3202-40fc-b4db-a9f7f643d17b-kube-api-access-p24rt" (OuterVolumeSpecName: "kube-api-access-p24rt") pod "18658936-3202-40fc-b4db-a9f7f643d17b" (UID: "18658936-3202-40fc-b4db-a9f7f643d17b"). InnerVolumeSpecName "kube-api-access-p24rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:41.433241 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.433204 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:41.433614 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.433600 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18658936-3202-40fc-b4db-a9f7f643d17b" containerName="test-curl-7696-140533220988080" Jul 21 15:04:41.433653 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.433615 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="18658936-3202-40fc-b4db-a9f7f643d17b" containerName="test-curl-7696-140533220988080" Jul 21 15:04:41.433699 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.433690 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="18658936-3202-40fc-b4db-a9f7f643d17b" containerName="test-curl-7696-140533220988080" Jul 21 15:04:41.436652 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.436637 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:41.444729 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.444706 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:41.462544 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.462514 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsxs\" (UniqueName: \"kubernetes.io/projected/107d9f8d-6221-4809-bb5a-a47f5a6eedb2-kube-api-access-mvsxs\") pod \"test-curl-7696-140533220989088\" (UID: \"107d9f8d-6221-4809-bb5a-a47f5a6eedb2\") " pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:41.462695 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.462577 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24rt\" (UniqueName: \"kubernetes.io/projected/18658936-3202-40fc-b4db-a9f7f643d17b-kube-api-access-p24rt\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:41.563652 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.563602 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsxs\" (UniqueName: \"kubernetes.io/projected/107d9f8d-6221-4809-bb5a-a47f5a6eedb2-kube-api-access-mvsxs\") pod \"test-curl-7696-140533220989088\" (UID: \"107d9f8d-6221-4809-bb5a-a47f5a6eedb2\") " pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:41.576574 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.576543 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsxs\" (UniqueName: \"kubernetes.io/projected/107d9f8d-6221-4809-bb5a-a47f5a6eedb2-kube-api-access-mvsxs\") pod \"test-curl-7696-140533220989088\" (UID: \"107d9f8d-6221-4809-bb5a-a47f5a6eedb2\") " pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:41.746488 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.746451 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:41.889374 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:41.889348 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:41.890960 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:41.890936 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod107d9f8d_6221_4809_bb5a_a47f5a6eedb2.slice/crio-49dd29a8967b46ef420a3d088609efe90234b71b686ae6fd6c8a0ef2be9712ae WatchSource:0}: Error finding container 49dd29a8967b46ef420a3d088609efe90234b71b686ae6fd6c8a0ef2be9712ae: Status 404 returned error can't find the container with id 49dd29a8967b46ef420a3d088609efe90234b71b686ae6fd6c8a0ef2be9712ae Jul 21 15:04:42.120862 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:42.120781 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:42.120862 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:42.120804 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61743f72012e8445c7da985a765fa70509b1067faca59f6087a5f5f628f6be8c" Jul 21 15:04:42.122024 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:42.121990 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220989088" event={"ID":"107d9f8d-6221-4809-bb5a-a47f5a6eedb2","Type":"ContainerStarted","Data":"49dd29a8967b46ef420a3d088609efe90234b71b686ae6fd6c8a0ef2be9712ae"} Jul 21 15:04:42.131892 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:04:42.131863 2570 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-140533220988080\" is forbidden: User \"system:node:ip-10-0-134-49.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-49.ec2.internal' and this object" podUID="18658936-3202-40fc-b4db-a9f7f643d17b" pod="opendatahub/test-curl-7696-140533220988080" Jul 21 15:04:42.140968 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:42.140943 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18658936-3202-40fc-b4db-a9f7f643d17b" path="/var/lib/kubelet/pods/18658936-3202-40fc-b4db-a9f7f643d17b/volumes" Jul 21 15:04:43.129566 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:43.129529 2570 generic.go:356] "Generic (PLEG): container finished" podID="107d9f8d-6221-4809-bb5a-a47f5a6eedb2" containerID="3d018fd975bde8cb5feca9b2e96df30200b233cf93e87638428807637f75462c" exitCode=0 Jul 21 15:04:43.129956 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:43.129593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220989088" event={"ID":"107d9f8d-6221-4809-bb5a-a47f5a6eedb2","Type":"ContainerDied","Data":"3d018fd975bde8cb5feca9b2e96df30200b233cf93e87638428807637f75462c"} Jul 21 15:04:43.826632 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:43.826576 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" podUID="b63c2838-d5a9-40e7-a865-1b45061a93ba" containerName="main" containerID="cri-o://a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993" gracePeriod=45 Jul 21 15:04:44.087174 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.087116 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:04:44.136598 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.136559 2570 generic.go:356] "Generic (PLEG): container finished" podID="b63c2838-d5a9-40e7-a865-1b45061a93ba" containerID="a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993" exitCode=0 Jul 21 15:04:44.136948 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.136659 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" Jul 21 15:04:44.142961 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.142932 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" event={"ID":"b63c2838-d5a9-40e7-a865-1b45061a93ba","Type":"ContainerDied","Data":"a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993"} Jul 21 15:04:44.143100 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.142973 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k" event={"ID":"b63c2838-d5a9-40e7-a865-1b45061a93ba","Type":"ContainerDied","Data":"1189a9d026ae66bab6a0ea2d490cd3e822cfc6494aa09e5d9a29fbae846616ca"} Jul 21 15:04:44.143100 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.142991 2570 scope.go:117] "RemoveContainer" containerID="a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993" Jul 21 15:04:44.152584 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.152564 2570 scope.go:117] "RemoveContainer" containerID="a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993" Jul 21 15:04:44.152873 ip-10-0-134-49 kubenswrapper[2570]: E0721 15:04:44.152853 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993\": container with ID starting with a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993 not found: ID does not exist" containerID="a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993" Jul 21 15:04:44.152930 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.152881 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993"} err="failed to get container status \"a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993\": rpc error: code = NotFound desc = could not find container \"a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993\": container with ID starting with a1a9adaee96523332c922bb3ac0262173c6607d03486c86650782b0fb46a3993 not found: ID does not exist" Jul 21 15:04:44.187721 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.187692 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlwqk\" (UniqueName: \"kubernetes.io/projected/b63c2838-d5a9-40e7-a865-1b45061a93ba-kube-api-access-tlwqk\") pod \"b63c2838-d5a9-40e7-a865-1b45061a93ba\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " Jul 21 15:04:44.187897 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.187740 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-model-cache\") pod \"b63c2838-d5a9-40e7-a865-1b45061a93ba\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " Jul 21 15:04:44.187897 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.187799 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-home\") pod \"b63c2838-d5a9-40e7-a865-1b45061a93ba\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " Jul 21 15:04:44.187897 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.187845 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-dshm\") pod \"b63c2838-d5a9-40e7-a865-1b45061a93ba\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " Jul 21 15:04:44.187897 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.187870 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63c2838-d5a9-40e7-a865-1b45061a93ba-tls-certs\") pod \"b63c2838-d5a9-40e7-a865-1b45061a93ba\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " Jul 21 15:04:44.188139 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.187912 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-tmp-dir\") pod \"b63c2838-d5a9-40e7-a865-1b45061a93ba\" (UID: \"b63c2838-d5a9-40e7-a865-1b45061a93ba\") " Jul 21 15:04:44.188139 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.188076 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "b63c2838-d5a9-40e7-a865-1b45061a93ba" (UID: "b63c2838-d5a9-40e7-a865-1b45061a93ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:04:44.188139 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.188128 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-home" (OuterVolumeSpecName: "home") pod "b63c2838-d5a9-40e7-a865-1b45061a93ba" (UID: "b63c2838-d5a9-40e7-a865-1b45061a93ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:04:44.188282 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.188233 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-model-cache\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.188282 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.188254 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-home\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.188282 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.188274 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b63c2838-d5a9-40e7-a865-1b45061a93ba" (UID: "b63c2838-d5a9-40e7-a865-1b45061a93ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:04:44.190317 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.190290 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-dshm" (OuterVolumeSpecName: "dshm") pod "b63c2838-d5a9-40e7-a865-1b45061a93ba" (UID: "b63c2838-d5a9-40e7-a865-1b45061a93ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 15:04:44.190317 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.190298 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63c2838-d5a9-40e7-a865-1b45061a93ba-kube-api-access-tlwqk" (OuterVolumeSpecName: "kube-api-access-tlwqk") pod "b63c2838-d5a9-40e7-a865-1b45061a93ba" (UID: "b63c2838-d5a9-40e7-a865-1b45061a93ba"). InnerVolumeSpecName "kube-api-access-tlwqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:44.190819 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.190784 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b63c2838-d5a9-40e7-a865-1b45061a93ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b63c2838-d5a9-40e7-a865-1b45061a93ba" (UID: "b63c2838-d5a9-40e7-a865-1b45061a93ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 15:04:44.263824 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.263795 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:44.289344 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.289308 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlwqk\" (UniqueName: \"kubernetes.io/projected/b63c2838-d5a9-40e7-a865-1b45061a93ba-kube-api-access-tlwqk\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.289344 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.289337 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-dshm\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.289344 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.289350 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b63c2838-d5a9-40e7-a865-1b45061a93ba-tls-certs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.289573 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.289388 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b63c2838-d5a9-40e7-a865-1b45061a93ba-tmp-dir\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.307105 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.307079 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-140533220989088_107d9f8d-6221-4809-bb5a-a47f5a6eedb2/test-curl-7696-140533220989088/0.log" Jul 21 15:04:44.333120 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.333083 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:44.341348 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.341280 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:44.390310 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.390277 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvsxs\" (UniqueName: \"kubernetes.io/projected/107d9f8d-6221-4809-bb5a-a47f5a6eedb2-kube-api-access-mvsxs\") pod \"107d9f8d-6221-4809-bb5a-a47f5a6eedb2\" (UID: \"107d9f8d-6221-4809-bb5a-a47f5a6eedb2\") " Jul 21 15:04:44.392287 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.392260 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107d9f8d-6221-4809-bb5a-a47f5a6eedb2-kube-api-access-mvsxs" (OuterVolumeSpecName: "kube-api-access-mvsxs") pod "107d9f8d-6221-4809-bb5a-a47f5a6eedb2" (UID: "107d9f8d-6221-4809-bb5a-a47f5a6eedb2"). InnerVolumeSpecName "kube-api-access-mvsxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:44.464660 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.464614 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:44.465042 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.465028 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107d9f8d-6221-4809-bb5a-a47f5a6eedb2" containerName="test-curl-7696-140533220989088" Jul 21 15:04:44.465042 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.465042 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="107d9f8d-6221-4809-bb5a-a47f5a6eedb2" containerName="test-curl-7696-140533220989088" Jul 21 15:04:44.465156 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.465060 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63c2838-d5a9-40e7-a865-1b45061a93ba" containerName="main" Jul 21 15:04:44.465156 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.465066 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63c2838-d5a9-40e7-a865-1b45061a93ba" containerName="main" Jul 21 15:04:44.465156 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.465123 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63c2838-d5a9-40e7-a865-1b45061a93ba" containerName="main" Jul 21 15:04:44.465156 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.465133 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="107d9f8d-6221-4809-bb5a-a47f5a6eedb2" containerName="test-curl-7696-140533220989088" Jul 21 15:04:44.468212 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.468191 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:44.468308 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.468251 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k"] Jul 21 15:04:44.474026 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.473978 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-449411/rate-test-model-449411-kserve-584df79dff-w5n5k"] Jul 21 15:04:44.480987 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.480963 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:44.491753 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.491727 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvsxs\" (UniqueName: \"kubernetes.io/projected/107d9f8d-6221-4809-bb5a-a47f5a6eedb2-kube-api-access-mvsxs\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:44.592260 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.592153 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtj9\" (UniqueName: \"kubernetes.io/projected/928e7da7-2bc6-4a89-928b-cb1c27b661de-kube-api-access-2xtj9\") pod \"test-curl-7696-140533220989088\" (UID: \"928e7da7-2bc6-4a89-928b-cb1c27b661de\") " pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:44.693368 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.693331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtj9\" (UniqueName: \"kubernetes.io/projected/928e7da7-2bc6-4a89-928b-cb1c27b661de-kube-api-access-2xtj9\") pod \"test-curl-7696-140533220989088\" (UID: \"928e7da7-2bc6-4a89-928b-cb1c27b661de\") " pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:44.706468 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.706435 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtj9\" (UniqueName: \"kubernetes.io/projected/928e7da7-2bc6-4a89-928b-cb1c27b661de-kube-api-access-2xtj9\") pod \"test-curl-7696-140533220989088\" (UID: \"928e7da7-2bc6-4a89-928b-cb1c27b661de\") " pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:44.777660 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.777616 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:44.922243 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:44.922219 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:44.924291 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:44.924264 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod928e7da7_2bc6_4a89_928b_cb1c27b661de.slice/crio-37a370881bb63d96ec3102b835dd3e4218c4a73e661bc2b1851dd97b17eb3886 WatchSource:0}: Error finding container 37a370881bb63d96ec3102b835dd3e4218c4a73e661bc2b1851dd97b17eb3886: Status 404 returned error can't find the container with id 37a370881bb63d96ec3102b835dd3e4218c4a73e661bc2b1851dd97b17eb3886 Jul 21 15:04:45.143133 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:45.143099 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220989088" event={"ID":"928e7da7-2bc6-4a89-928b-cb1c27b661de","Type":"ContainerStarted","Data":"37a370881bb63d96ec3102b835dd3e4218c4a73e661bc2b1851dd97b17eb3886"} Jul 21 15:04:45.145220 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:45.145200 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49dd29a8967b46ef420a3d088609efe90234b71b686ae6fd6c8a0ef2be9712ae" Jul 21 15:04:45.145351 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:45.145237 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:45.158901 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:45.158873 2570 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7696-140533220989088" oldPodUID="107d9f8d-6221-4809-bb5a-a47f5a6eedb2" podUID="928e7da7-2bc6-4a89-928b-cb1c27b661de" Jul 21 15:04:46.141336 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:46.141303 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="107d9f8d-6221-4809-bb5a-a47f5a6eedb2" path="/var/lib/kubelet/pods/107d9f8d-6221-4809-bb5a-a47f5a6eedb2/volumes" Jul 21 15:04:46.141703 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:46.141681 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b63c2838-d5a9-40e7-a865-1b45061a93ba" path="/var/lib/kubelet/pods/b63c2838-d5a9-40e7-a865-1b45061a93ba/volumes" Jul 21 15:04:46.150845 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:46.150817 2570 generic.go:356] "Generic (PLEG): container finished" podID="928e7da7-2bc6-4a89-928b-cb1c27b661de" containerID="219fed86df639a5de0044b44882b5291b5eb308d63907f49ef8e3aa271084947" exitCode=0 Jul 21 15:04:46.151195 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:46.150851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-140533220989088" event={"ID":"928e7da7-2bc6-4a89-928b-cb1c27b661de","Type":"ContainerDied","Data":"219fed86df639a5de0044b44882b5291b5eb308d63907f49ef8e3aa271084947"} Jul 21 15:04:47.314400 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.314369 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:47.368602 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.368571 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7696-140533220989088_928e7da7-2bc6-4a89-928b-cb1c27b661de/test-curl-7696-140533220989088/0.log" Jul 21 15:04:47.408498 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.408411 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:47.420855 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.420824 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-140533220989088"] Jul 21 15:04:47.424879 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.424860 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xtj9\" (UniqueName: \"kubernetes.io/projected/928e7da7-2bc6-4a89-928b-cb1c27b661de-kube-api-access-2xtj9\") pod \"928e7da7-2bc6-4a89-928b-cb1c27b661de\" (UID: \"928e7da7-2bc6-4a89-928b-cb1c27b661de\") " Jul 21 15:04:47.426891 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.426865 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928e7da7-2bc6-4a89-928b-cb1c27b661de-kube-api-access-2xtj9" (OuterVolumeSpecName: "kube-api-access-2xtj9") pod "928e7da7-2bc6-4a89-928b-cb1c27b661de" (UID: "928e7da7-2bc6-4a89-928b-cb1c27b661de"). InnerVolumeSpecName "kube-api-access-2xtj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 15:04:47.526410 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.526373 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtj9\" (UniqueName: \"kubernetes.io/projected/928e7da7-2bc6-4a89-928b-cb1c27b661de-kube-api-access-2xtj9\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:47.660179 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.660088 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6"] Jul 21 15:04:47.660522 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.660508 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928e7da7-2bc6-4a89-928b-cb1c27b661de" containerName="test-curl-7696-140533220989088" Jul 21 15:04:47.660522 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.660522 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="928e7da7-2bc6-4a89-928b-cb1c27b661de" containerName="test-curl-7696-140533220989088" Jul 21 15:04:47.660642 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.660604 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="928e7da7-2bc6-4a89-928b-cb1c27b661de" containerName="test-curl-7696-140533220989088" Jul 21 15:04:47.663740 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.663716 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:47.667924 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.667904 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 15:04:47.681942 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.681913 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6"] Jul 21 15:04:47.728888 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.728854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d376-cf5a-424c-8616-271b41e47449-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d89949-qntv6\" (UID: \"8535d376-cf5a-424c-8616-271b41e47449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:47.829946 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.829908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d376-cf5a-424c-8616-271b41e47449-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d89949-qntv6\" (UID: \"8535d376-cf5a-424c-8616-271b41e47449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:47.830657 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.830634 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d376-cf5a-424c-8616-271b41e47449-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d89949-qntv6\" (UID: \"8535d376-cf5a-424c-8616-271b41e47449\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:47.974404 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:47.974314 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:48.122217 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.122186 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6"] Jul 21 15:04:48.124603 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:04:48.124579 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8535d376_cf5a_424c_8616_271b41e47449.slice/crio-485b67cb114149bd52afa5cb3880e9e309115c37587fe81d15305fad4270030b WatchSource:0}: Error finding container 485b67cb114149bd52afa5cb3880e9e309115c37587fe81d15305fad4270030b: Status 404 returned error can't find the container with id 485b67cb114149bd52afa5cb3880e9e309115c37587fe81d15305fad4270030b Jul 21 15:04:48.142550 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.142528 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="928e7da7-2bc6-4a89-928b-cb1c27b661de" path="/var/lib/kubelet/pods/928e7da7-2bc6-4a89-928b-cb1c27b661de/volumes" Jul 21 15:04:48.161795 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.161776 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-140533220989088" Jul 21 15:04:48.161795 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.161794 2570 scope.go:117] "RemoveContainer" containerID="219fed86df639a5de0044b44882b5291b5eb308d63907f49ef8e3aa271084947" Jul 21 15:04:48.162945 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.162921 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" event={"ID":"8535d376-cf5a-424c-8616-271b41e47449","Type":"ContainerStarted","Data":"485b67cb114149bd52afa5cb3880e9e309115c37587fe81d15305fad4270030b"} Jul 21 15:04:48.704793 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.704738 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:48.808209 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:48.808171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:49.170355 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:49.170326 2570 generic.go:356] "Generic (PLEG): container finished" podID="8535d376-cf5a-424c-8616-271b41e47449" containerID="487bdccaf56cc184297fafeb985cfd93b260c868beba3252396e9c315377d798" exitCode=0 Jul 21 15:04:49.170526 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:49.170406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" event={"ID":"8535d376-cf5a-424c-8616-271b41e47449","Type":"ContainerDied","Data":"487bdccaf56cc184297fafeb985cfd93b260c868beba3252396e9c315377d798"} Jul 21 15:04:50.316428 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:50.316402 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:50.454228 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:50.454137 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d376-cf5a-424c-8616-271b41e47449-maas-api-service-ca\") pod \"8535d376-cf5a-424c-8616-271b41e47449\" (UID: \"8535d376-cf5a-424c-8616-271b41e47449\") " Jul 21 15:04:50.454512 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:50.454487 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8535d376-cf5a-424c-8616-271b41e47449-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8535d376-cf5a-424c-8616-271b41e47449" (UID: "8535d376-cf5a-424c-8616-271b41e47449"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:04:50.555400 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:50.555358 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8535d376-cf5a-424c-8616-271b41e47449-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:04:51.183326 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:51.183296 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" Jul 21 15:04:51.183504 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:51.183296 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6" event={"ID":"8535d376-cf5a-424c-8616-271b41e47449","Type":"ContainerDied","Data":"485b67cb114149bd52afa5cb3880e9e309115c37587fe81d15305fad4270030b"} Jul 21 15:04:51.183504 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:51.183421 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485b67cb114149bd52afa5cb3880e9e309115c37587fe81d15305fad4270030b" Jul 21 15:04:58.672210 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:58.672164 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:04:58.677814 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:58.677782 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc"] Jul 21 15:04:58.680688 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:04:58.680662 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d89949-29744100-mnmxc"] Jul 21 15:05:00.141406 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:00.141374 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2129b578-d97d-4b05-a0ad-8fdfbeb8d440" path="/var/lib/kubelet/pods/2129b578-d97d-4b05-a0ad-8fdfbeb8d440/volumes" Jul 21 15:05:02.812851 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:02.812795 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:02.846829 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:02.846796 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6"] Jul 21 15:05:02.853913 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:02.853887 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d89949-qntv6"] Jul 21 15:05:03.227801 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:03.227766 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:04.142137 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:04.142104 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8535d376-cf5a-424c-8616-271b41e47449" path="/var/lib/kubelet/pods/8535d376-cf5a-424c-8616-271b41e47449/volumes" Jul 21 15:05:10.275211 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.275170 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:10.349786 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.349749 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:10.418558 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.418525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:10.599190 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.599089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:10.721472 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.721442 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz"] Jul 21 15:05:10.721914 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.721896 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8535d376-cf5a-424c-8616-271b41e47449" containerName="revoke-keys" Jul 21 15:05:10.721999 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.721916 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8535d376-cf5a-424c-8616-271b41e47449" containerName="revoke-keys" Jul 21 15:05:10.722076 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.722026 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8535d376-cf5a-424c-8616-271b41e47449" containerName="revoke-keys" Jul 21 15:05:10.726938 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.726919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:10.730133 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.730116 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mfc8l\"" Jul 21 15:05:10.734680 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.734654 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz"] Jul 21 15:05:10.832697 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.832672 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-449411-sq6sz\" (UID: \"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:10.933309 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.933286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-449411-sq6sz\" (UID: \"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:10.933819 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:10.933803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-449411-sq6sz\" (UID: \"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:11.038647 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:11.038627 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:11.180706 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:11.180679 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz"] Jul 21 15:05:11.182618 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:05:11.182592 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e13cd1e_5005_4e99_ba76_ad2f15a30f8c.slice/crio-b877c563ec0c8d26b690c2370016ab438d316e3aee4a5551670cc8a82a32f378 WatchSource:0}: Error finding container b877c563ec0c8d26b690c2370016ab438d316e3aee4a5551670cc8a82a32f378: Status 404 returned error can't find the container with id b877c563ec0c8d26b690c2370016ab438d316e3aee4a5551670cc8a82a32f378 Jul 21 15:05:11.292489 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:11.292464 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" event={"ID":"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c","Type":"ContainerStarted","Data":"b877c563ec0c8d26b690c2370016ab438d316e3aee4a5551670cc8a82a32f378"} Jul 21 15:05:11.645248 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:11.645195 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:12.046546 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:12.046477 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:12.298895 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:12.298813 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e13cd1e-5005-4e99-ba76-ad2f15a30f8c" containerID="dadfcf233d70ce00224da4102da53cb3d0a2e70fbfe8287f1c469224ab1fa5f1" exitCode=0 Jul 21 15:05:12.298895 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:12.298860 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" event={"ID":"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c","Type":"ContainerDied","Data":"dadfcf233d70ce00224da4102da53cb3d0a2e70fbfe8287f1c469224ab1fa5f1"} Jul 21 15:05:13.443628 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:13.443603 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:13.450705 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:13.450683 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c-maas-api-service-ca\") pod \"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c\" (UID: \"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c\") " Jul 21 15:05:13.450956 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:13.450935 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e13cd1e-5005-4e99-ba76-ad2f15a30f8c" (UID: "7e13cd1e-5005-4e99-ba76-ad2f15a30f8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 15:05:13.551313 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:13.551288 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c-maas-api-service-ca\") on node \"ip-10-0-134-49.ec2.internal\" DevicePath \"\"" Jul 21 15:05:14.310662 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:14.310636 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" Jul 21 15:05:14.310800 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:14.310636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz" event={"ID":"7e13cd1e-5005-4e99-ba76-ad2f15a30f8c","Type":"ContainerDied","Data":"b877c563ec0c8d26b690c2370016ab438d316e3aee4a5551670cc8a82a32f378"} Jul 21 15:05:14.310800 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:14.310759 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b877c563ec0c8d26b690c2370016ab438d316e3aee4a5551670cc8a82a32f378" Jul 21 15:05:21.408399 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:21.408363 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:21.415776 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:21.415743 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q"] Jul 21 15:05:21.417667 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:21.417649 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-449411-29744100-pqn9q"] Jul 21 15:05:22.141230 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:22.141190 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae0ee80-1772-4663-9ec1-aed5e1d2e202" path="/var/lib/kubelet/pods/7ae0ee80-1772-4663-9ec1-aed5e1d2e202/volumes" Jul 21 15:05:25.838420 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:25.838387 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz"] Jul 21 15:05:25.841386 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:25.841362 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-449411-sq6sz"] Jul 21 15:05:26.123136 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:26.123068 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:26.140876 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:26.140850 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e13cd1e-5005-4e99-ba76-ad2f15a30f8c" path="/var/lib/kubelet/pods/7e13cd1e-5005-4e99-ba76-ad2f15a30f8c/volumes" Jul 21 15:05:26.507231 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:26.507196 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:33.625459 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:33.625418 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:34.144275 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:34.144243 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:34.246682 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:34.246652 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:34.378334 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:34.378313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:40.470414 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:40.470381 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-trj7w_43c250ff-e40d-49fd-94cf-d66c1f7c7960/manager/0.log" Jul 21 15:05:40.735988 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:40.735910 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84b95d7b79-45qnw_76522a94-1e17-4d4c-9107-003dc76df617/manager/0.log" Jul 21 15:05:40.922948 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:40.922922 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dswlc_874af69e-95c2-4b04-8124-fa68f9a0e8ce/server/0.log" Jul 21 15:05:41.051825 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:41.051757 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z4zgl_216dd99e-221f-4ab3-b954-d7fc4f43954c/manager/0.log" Jul 21 15:05:41.177495 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:41.177469 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bb54d755-6wlhn_1224731d-6898-403b-9a72-061884710a78/copy-manifests/0.log" Jul 21 15:05:41.199378 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:41.199352 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bb54d755-6wlhn_1224731d-6898-403b-9a72-061884710a78/manager/0.log" Jul 21 15:05:41.838103 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:41.838063 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85cb597c75-ld6th_c3981ae9-d510-40e8-b056-9c3f963afbf4/maas-api/0.log" Jul 21 15:05:41.954798 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:41.954770 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744070-kjs8g_f9f6f889-7dbc-4e58-bb13-07551c0abafd/cleanup/0.log" Jul 21 15:05:42.066094 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:42.066073 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744085-wxszp_29c9b1d4-6843-4ab4-a3a8-21d6323b9596/cleanup/0.log" Jul 21 15:05:42.186484 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:42.186443 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744100-vc57l_2ed9c11a-3b15-48ad-a824-dbdf9dde7715/cleanup/0.log" Jul 21 15:05:42.303601 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:42.303564 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-vdcnl_40624fc1-938e-4096-b24a-0e57bd4de01f/postgres/0.log" Jul 21 15:05:43.734574 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:43.734542 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hrx5d_ae07fb97-090c-4368-a75d-20676aa30d08/manager/0.log" Jul 21 15:05:43.853914 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:43.853882 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-82pkr_e31ff0c9-264c-4660-8936-a3d35c3dde6c/manager/0.log" Jul 21 15:05:44.089907 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:44.089830 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6bpsr_c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a/registry-server/0.log" Jul 21 15:05:44.222882 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:44.222849 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xls52_32aa0b2a-a811-47eb-8f26-6e5a77bb0b17/manager/0.log" Jul 21 15:05:44.471625 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:44.471594 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-r7tsl_c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc/manager/0.log" Jul 21 15:05:45.199338 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:45.199310 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd655967-vc576_d1079e15-b066-49c9-9476-a14d12ff7182/kube-auth-proxy/0.log" Jul 21 15:05:45.572744 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:45.572657 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sr9z2_b45e7a32-477d-4bf5-b08d-6c2c43d0e266/payload-processing/0.log" Jul 21 15:05:45.690359 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:45.690334 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7bf988cb-t286m_d50910e2-559c-4954-b35e-d877abde0ef4/router/0.log" Jul 21 15:05:46.179134 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:46.179107 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-spdxm_9e478ed3-a528-428e-bb26-41aa712be148/main/0.log" Jul 21 15:05:46.422769 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:46.422739 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fdnff4_de6587b4-b048-4c66-bbb4-47f5d2d0b3cc/main/0.log" Jul 21 15:05:53.787312 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:53.787275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:54.130965 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:54.130879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" limit=1024 Jul 21 15:05:57.369894 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:57.369860 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p447f_1f9a0b12-10a7-4628-92ed-9d9651940451/global-pull-secret-syncer/0.log" Jul 21 15:05:57.432183 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:57.432158 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9725k_90b15efc-d945-4740-9546-757b99ec15d2/konnectivity-agent/0.log" Jul 21 15:05:57.579053 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:05:57.579032 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-49.ec2.internal_62ca7a90d9db4064d93d60a128f27362/haproxy/0.log" Jul 21 15:06:03.288899 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:03.288858 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hrx5d_ae07fb97-090c-4368-a75d-20676aa30d08/manager/0.log" Jul 21 15:06:03.321949 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:03.321923 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-82pkr_e31ff0c9-264c-4660-8936-a3d35c3dde6c/manager/0.log" Jul 21 15:06:03.432465 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:03.432438 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6bpsr_c0f31b9e-5ef3-4926-a318-b9cc7cf1d45a/registry-server/0.log" Jul 21 15:06:03.611724 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:03.611641 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-xls52_32aa0b2a-a811-47eb-8f26-6e5a77bb0b17/manager/0.log" Jul 21 15:06:03.760729 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:03.760678 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-r7tsl_c5f78da9-3ed0-4f90-82d7-c64eeb4f5bcc/manager/0.log" Jul 21 15:06:05.788549 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:05.788519 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-fxl5n_d18d8dde-86bd-4fff-8a27-a0c0280720a4/cluster-monitoring-operator/0.log" Jul 21 15:06:06.229067 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:06.229035 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4dmd_5bf3f586-f2c5-43a5-a749-96e81395e317/node-exporter/0.log" Jul 21 15:06:06.257126 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:06.257101 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4dmd_5bf3f586-f2c5-43a5-a749-96e81395e317/kube-rbac-proxy/0.log" Jul 21 15:06:06.291074 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:06.291056 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r4dmd_5bf3f586-f2c5-43a5-a749-96e81395e317/init-textfile/0.log" Jul 21 15:06:08.044217 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.044182 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q"] Jul 21 15:06:08.044635 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.044621 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e13cd1e-5005-4e99-ba76-ad2f15a30f8c" containerName="revoke-keys" Jul 21 15:06:08.044681 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.044638 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e13cd1e-5005-4e99-ba76-ad2f15a30f8c" containerName="revoke-keys" Jul 21 15:06:08.044740 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.044728 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e13cd1e-5005-4e99-ba76-ad2f15a30f8c" containerName="revoke-keys" Jul 21 15:06:08.047986 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.047969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.051646 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.051618 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5jhqw\"/\"default-dockercfg-lz5ll\"" Jul 21 15:06:08.051768 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.051626 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jhqw\"/\"kube-root-ca.crt\"" Jul 21 15:06:08.051768 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.051672 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jhqw\"/\"openshift-service-ca.crt\"" Jul 21 15:06:08.065018 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.064987 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q"] Jul 21 15:06:08.175179 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.175156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-sys\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.175306 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.175191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-lib-modules\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.175306 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.175212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjzj\" (UniqueName: \"kubernetes.io/projected/5420281e-4d9f-466d-9c49-17cde13a5070-kube-api-access-2vjzj\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.175421 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.175302 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-proc\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.175421 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.175329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-podres\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.175639 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.175622 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-jvz68_bf728902-c923-4e7a-aeb1-94338a8b1d95/networking-console-plugin/0.log" Jul 21 15:06:08.276135 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276109 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-proc\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276266 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276148 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-podres\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276266 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276227 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-proc\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276266 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-sys\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276411 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-lib-modules\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276411 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276294 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-sys\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276411 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjzj\" (UniqueName: \"kubernetes.io/projected/5420281e-4d9f-466d-9c49-17cde13a5070-kube-api-access-2vjzj\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276411 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276341 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-podres\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.276535 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.276446 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5420281e-4d9f-466d-9c49-17cde13a5070-lib-modules\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.287576 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.287548 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjzj\" (UniqueName: \"kubernetes.io/projected/5420281e-4d9f-466d-9c49-17cde13a5070-kube-api-access-2vjzj\") pod \"perf-node-gather-daemonset-gm22q\" (UID: \"5420281e-4d9f-466d-9c49-17cde13a5070\") " pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.358228 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.358156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.491427 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.491400 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q"] Jul 21 15:06:08.493822 ip-10-0-134-49 kubenswrapper[2570]: W0721 15:06:08.493790 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5420281e_4d9f_466d_9c49_17cde13a5070.slice/crio-d843f22658de36db31a5f17a9f6b7c7a6974000cda525b1197c8056acba3840b WatchSource:0}: Error finding container d843f22658de36db31a5f17a9f6b7c7a6974000cda525b1197c8056acba3840b: Status 404 returned error can't find the container with id d843f22658de36db31a5f17a9f6b7c7a6974000cda525b1197c8056acba3840b Jul 21 15:06:08.604240 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.604217 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" event={"ID":"5420281e-4d9f-466d-9c49-17cde13a5070","Type":"ContainerStarted","Data":"8617116d748574893941992f2b688cd5e098fd4cc6931295834bb5fe487a8cf4"} Jul 21 15:06:08.604357 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.604259 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" event={"ID":"5420281e-4d9f-466d-9c49-17cde13a5070","Type":"ContainerStarted","Data":"d843f22658de36db31a5f17a9f6b7c7a6974000cda525b1197c8056acba3840b"} Jul 21 15:06:08.604357 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.604298 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:08.625663 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.625572 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" podStartSLOduration=0.625560165 podStartE2EDuration="625.560165ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 15:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 15:06:08.624230239 +0000 UTC m=+2930.991740266" watchObservedRunningTime="2026-07-21 15:06:08.625560165 +0000 UTC m=+2930.993070193" Jul 21 15:06:08.763636 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.763617 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/2.log" Jul 21 15:06:08.772930 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:08.772911 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-kg4q9_f7f3c68c-9553-4b61-a33f-c30d4e11f9c1/console-operator/3.log" Jul 21 15:06:09.975549 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:09.975509 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-pq8nz_5fd4ceb5-a5e6-4cb4-890a-03ff64d2c2dd/volume-data-source-validator/0.log" Jul 21 15:06:11.164623 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:11.164574 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76pfl_e444c9d9-48ba-4131-9b16-2c8e26296df0/dns/0.log" Jul 21 15:06:11.194473 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:11.194442 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-76pfl_e444c9d9-48ba-4131-9b16-2c8e26296df0/kube-rbac-proxy/0.log" Jul 21 15:06:11.440101 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:11.440022 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g442s_292c96a4-7f66-4fca-adfc-8a86176f342a/dns-node-resolver/0.log" Jul 21 15:06:12.153722 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:12.153666 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7t9g8_f91b710d-ac22-48bd-b037-439391d77258/node-ca/0.log" Jul 21 15:06:13.454378 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:13.454344 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd655967-vc576_d1079e15-b066-49c9-9476-a14d12ff7182/kube-auth-proxy/0.log" Jul 21 15:06:13.589014 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:13.588986 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sr9z2_b45e7a32-477d-4bf5-b08d-6c2c43d0e266/payload-processing/0.log" Jul 21 15:06:13.632816 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:13.632790 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7bf988cb-t286m_d50910e2-559c-4954-b35e-d877abde0ef4/router/0.log" Jul 21 15:06:14.452153 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:14.452117 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xp7xl_d9f27890-994f-491e-b14e-52319993fc85/serve-healthcheck-canary/0.log" Jul 21 15:06:14.619519 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:14.619495 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5jhqw/perf-node-gather-daemonset-gm22q" Jul 21 15:06:15.066565 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:15.066536 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kglk5_e6bdcbdf-1506-490a-bd18-64ec3f4ce72a/insights-operator/0.log" Jul 21 15:06:15.183307 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:15.183278 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v7457_a3e67ee5-39c5-4e35-94dc-b5db10c90996/kube-rbac-proxy/0.log" Jul 21 15:06:15.211170 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:15.211146 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v7457_a3e67ee5-39c5-4e35-94dc-b5db10c90996/exporter/0.log" Jul 21 15:06:15.238547 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:15.238525 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v7457_a3e67ee5-39c5-4e35-94dc-b5db10c90996/extractor/0.log" Jul 21 15:06:17.866817 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:17.866782 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-trj7w_43c250ff-e40d-49fd-94cf-d66c1f7c7960/manager/0.log" Jul 21 15:06:18.250558 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.250482 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd655967-vc576_d1079e15-b066-49c9-9476-a14d12ff7182/kube-auth-proxy/0.log" Jul 21 15:06:18.360445 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.360415 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sr9z2_b45e7a32-477d-4bf5-b08d-6c2c43d0e266/payload-processing/0.log" Jul 21 15:06:18.381309 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.381260 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7bf988cb-t286m_d50910e2-559c-4954-b35e-d877abde0ef4/router/0.log" Jul 21 15:06:18.430620 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.430598 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84b95d7b79-45qnw_76522a94-1e17-4d4c-9107-003dc76df617/manager/0.log" Jul 21 15:06:18.458640 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.458621 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dswlc_874af69e-95c2-4b04-8124-fa68f9a0e8ce/server/0.log" Jul 21 15:06:18.572927 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.572874 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z4zgl_216dd99e-221f-4ab3-b954-d7fc4f43954c/manager/0.log" Jul 21 15:06:18.577809 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.577789 2570 scope.go:117] "RemoveContainer" containerID="479871b2f65169f7f8199ac856d14d4c8e4c4edbe5d0e6f1827c404c0e201308" Jul 21 15:06:18.585977 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.585961 2570 scope.go:117] "RemoveContainer" containerID="9494aba8a20d2155fd146fec89483cdb11210f214acece1c4bb98b4a2f9a520a" Jul 21 15:06:18.712398 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.712368 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bb54d755-6wlhn_1224731d-6898-403b-9a72-061884710a78/manager/0.log" Jul 21 15:06:18.732090 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:18.732071 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86bb54d755-6wlhn_1224731d-6898-403b-9a72-061884710a78/copy-manifests/0.log" Jul 21 15:06:19.717816 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:19.717785 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5bbd655967-vc576_d1079e15-b066-49c9-9476-a14d12ff7182/kube-auth-proxy/0.log" Jul 21 15:06:19.837589 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:19.837493 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-sr9z2_b45e7a32-477d-4bf5-b08d-6c2c43d0e266/payload-processing/0.log" Jul 21 15:06:19.857316 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:19.857279 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d7bf988cb-t286m_d50910e2-559c-4954-b35e-d877abde0ef4/router/0.log" Jul 21 15:06:25.371859 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:25.371824 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-4fkls_4302d816-607c-4944-92ed-8618212a8ecb/migrator/0.log" Jul 21 15:06:25.399531 ip-10-0-134-49 kubenswrapper[2570]: I0721 15:06:25.399497 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-4fkls_4302d816-607c-4944-92ed-8618212a8ecb/graceful-termination/0.log"