Jul 24 00:08:39.972574 ip-10-0-136-102 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 00:08:39.972584 ip-10-0-136-102 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 00:08:39.972592 ip-10-0-136-102 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 00:08:39.972812 ip-10-0-136-102 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 00:08:50.022798 ip-10-0-136-102 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 00:08:50.022815 ip-10-0-136-102 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 305decd526b44ae8ad452d8ebd9bfc0e -- Jul 24 00:10:23.132246 ip-10-0-136-102 systemd[1]: Starting Kubernetes Kubelet... Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.565943 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570799 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570814 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570817 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570820 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570823 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570826 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570828 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:10:23.647626 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570832 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:10:23.643918 ip-10-0-136-102 systemd[1]: Started Kubernetes Kubelet. Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570835 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570838 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570840 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570843 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570845 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570849 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570852 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570855 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570875 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570879 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570881 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570885 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570888 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570891 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570894 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570897 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570899 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570902 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570905 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:10:23.669406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570908 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570910 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570913 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570915 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570918 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570921 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570923 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570926 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570929 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570931 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570934 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570939 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570941 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570944 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570947 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570949 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570952 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570955 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570957 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570961 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:10:23.873966 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570964 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570966 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570969 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570972 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570974 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570977 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570979 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570983 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570985 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570988 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570990 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570994 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570997 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.570999 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571002 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571006 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571011 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571016 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571019 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:10:23.883831 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571022 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571025 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571028 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571032 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571035 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571039 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571042 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571044 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571049 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571051 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571054 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571057 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571061 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571063 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571066 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571069 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571072 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571075 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571077 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571080 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:10:23.884715 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571082 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571085 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571087 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571090 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571092 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571095 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571098 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571100 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571104 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571107 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571110 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571112 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571115 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571882 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571890 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571893 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571895 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571898 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571902 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:10:23.885699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571906 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571909 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571912 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571915 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571917 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571920 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571923 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571926 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571929 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571932 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571935 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571939 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571941 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571944 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571947 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571951 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571953 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571956 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571959 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571961 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:10:23.886324 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571964 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571967 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571969 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571972 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571977 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571981 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571984 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571987 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571991 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571993 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571996 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.571998 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572006 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572009 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572012 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572014 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572018 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572020 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572024 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:10:23.887154 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572028 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572032 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572035 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572039 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572042 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572045 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572048 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572051 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572053 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572056 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572059 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572062 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572065 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572069 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572071 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572074 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572077 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572080 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572083 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:10:23.887772 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572085 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572088 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572091 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572093 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572098 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572101 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572104 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572107 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572110 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572113 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572115 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572118 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572120 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572123 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572125 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572128 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572131 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572133 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572136 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572139 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:10:23.888393 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572141 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572144 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572146 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572149 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572152 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572154 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572157 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572159 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572162 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572165 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572167 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572170 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572173 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572176 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572178 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572584 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572591 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572594 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572596 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:10:23.889049 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572602 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572605 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572608 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572610 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572613 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572615 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572618 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572621 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572623 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572626 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572628 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572631 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572634 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572637 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572640 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572642 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572646 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572649 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572651 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572654 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:10:23.889723 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572656 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572659 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572662 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572664 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572667 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572669 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572672 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572674 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572677 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572682 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572685 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572688 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572690 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572694 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572696 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572699 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572702 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572704 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572707 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572709 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572712 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:10:23.890340 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572714 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572717 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572719 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572724 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572728 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572731 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572734 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572738 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572743 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572747 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572750 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572752 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572755 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572758 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572761 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572764 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572767 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572771 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572774 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:10:23.891011 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572777 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572781 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572784 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572786 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572789 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572792 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572796 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572798 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572801 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572803 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572806 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572808 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572811 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572814 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572817 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572819 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572822 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572824 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572827 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572829 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:10:23.891515 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572833 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572835 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572838 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572840 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572843 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572845 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572848 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572850 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572853 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572856 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572878 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572881 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572884 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572889 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.572891 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.572968 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.572976 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.572982 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.572987 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.572993 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 00:10:23.892102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.572996 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573001 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573006 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573009 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573012 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573016 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573020 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573023 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573026 2585 flags.go:64] FLAG: --cgroup-root="" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573029 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573032 2585 flags.go:64] FLAG: --client-ca-file="" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573035 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573040 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573044 2585 flags.go:64] FLAG: --cluster-domain="" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573047 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573050 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573054 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573057 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573061 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573064 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573068 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573071 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573074 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573077 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 00:10:23.892630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573080 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573085 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573088 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573092 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573095 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573098 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573101 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573104 2585 flags.go:64] FLAG: --enable-server="true" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573108 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573113 2585 flags.go:64] FLAG: --event-burst="100" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573116 2585 flags.go:64] FLAG: --event-qps="50" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573119 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573122 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573125 2585 flags.go:64] FLAG: --eviction-hard="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573129 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573132 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573135 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573138 2585 flags.go:64] FLAG: --eviction-soft="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573141 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573144 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573148 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573152 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573155 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573158 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573161 2585 flags.go:64] FLAG: --feature-gates="" Jul 24 00:10:23.893378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573164 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573167 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573171 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573174 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573177 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573180 2585 flags.go:64] FLAG: --help="false" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573183 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-136-102.ec2.internal" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573186 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573191 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573194 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573198 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573201 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573205 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573208 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573211 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573214 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573218 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573221 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573224 2585 flags.go:64] FLAG: --kube-reserved="" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573227 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573230 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573233 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573236 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573239 2585 flags.go:64] FLAG: --lock-file="" Jul 24 00:10:23.894098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573242 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573245 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573248 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573255 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573258 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573261 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573264 2585 flags.go:64] FLAG: --logging-format="text" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573267 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573270 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573273 2585 flags.go:64] FLAG: --manifest-url="" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573276 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573281 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573284 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573288 2585 flags.go:64] FLAG: --max-pods="110" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573291 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573294 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573299 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573302 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573305 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573308 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573311 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573320 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573323 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573326 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 00:10:23.894854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573330 2585 flags.go:64] FLAG: --pod-cidr="" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573333 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573339 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573342 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573346 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573348 2585 flags.go:64] FLAG: --port="10250" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573351 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573419 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05d331e74626fe71f" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573426 2585 flags.go:64] FLAG: --qos-reserved="" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573431 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573435 2585 flags.go:64] FLAG: --register-node="true" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573439 2585 flags.go:64] FLAG: --register-with-taints="" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573444 2585 flags.go:64] FLAG: --registry-burst="10" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573448 2585 flags.go:64] FLAG: --registry-qps="5" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573451 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573454 2585 flags.go:64] FLAG: --reserved-memory="" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573458 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573462 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573465 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573468 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573473 2585 flags.go:64] FLAG: --runonce="false" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573476 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573480 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573484 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573487 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 00:10:23.895614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573490 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573494 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573498 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573502 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573505 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573508 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573511 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573514 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573517 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573521 2585 flags.go:64] FLAG: --system-cgroups="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573523 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573535 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573538 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573541 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573550 2585 flags.go:64] FLAG: --tls-min-version="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573559 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573562 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573565 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573568 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573578 2585 flags.go:64] FLAG: --v="2" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573584 2585 flags.go:64] FLAG: --version="false" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573589 2585 flags.go:64] FLAG: --vmodule="" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573593 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.573596 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573796 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:10:23.896466 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573801 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573804 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573807 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573810 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573813 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573815 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573818 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573822 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573824 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573828 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573831 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573834 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573837 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573839 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573842 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573845 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573849 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573856 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573872 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:10:23.898357 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573875 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573877 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573880 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573883 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573888 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573891 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573895 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573898 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573901 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573904 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573907 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573910 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573913 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573916 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573918 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573921 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573923 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573926 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573929 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:10:23.899531 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573931 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573934 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573936 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573939 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573942 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573944 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573947 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573950 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573953 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573957 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573960 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573963 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573966 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573969 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573971 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573974 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573977 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573980 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573983 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573986 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:10:24.014133 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573989 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573991 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573994 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573997 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.573999 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574002 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574004 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574007 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574010 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574013 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574016 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574018 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574021 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574023 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574026 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574028 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574031 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574033 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574036 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574039 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:10:24.014838 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574042 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574046 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574049 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574052 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574055 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574057 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574060 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574063 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574066 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574069 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574072 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574076 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574080 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574083 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574086 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574089 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574091 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574094 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574097 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:10:24.015438 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.574100 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.574694 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.584683 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.584705 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.585533 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.585542 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585615 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585621 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585625 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585629 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585631 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585635 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585637 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585641 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585643 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:10:24.015995 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585649 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585654 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585657 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585660 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585663 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585666 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585669 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585672 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585675 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585678 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585680 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585683 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585691 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585694 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585697 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585700 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585702 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585705 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585708 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585711 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:10:24.016451 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585714 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585716 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585719 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585721 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585724 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585727 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585729 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585732 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585734 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585738 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585740 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585744 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585749 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585754 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585757 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585760 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585763 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585767 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585770 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:10:24.017048 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585773 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585776 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585778 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585781 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585785 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585789 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585792 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585795 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585797 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585800 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585802 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585805 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585808 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585810 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585813 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585816 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585818 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585821 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585823 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:10:24.017550 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585826 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585829 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585832 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585834 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585837 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585840 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585844 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585848 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585851 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585853 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585856 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585874 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585878 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585881 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585884 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585887 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585890 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585893 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585895 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585898 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:10:24.018122 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585901 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585904 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585906 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585909 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585912 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585915 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585917 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585920 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585923 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585925 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585930 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.585933 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.585938 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586072 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 00:10:24.018684 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586078 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586081 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586084 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586087 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586092 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586098 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586102 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586104 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586107 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586110 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586113 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586117 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586120 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586123 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586126 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586128 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586131 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586133 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586136 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 00:10:24.019139 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586138 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586141 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586144 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586147 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586149 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586152 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586156 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586158 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586161 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586163 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586166 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586169 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586172 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586174 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586177 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586179 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586182 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586185 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586190 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586193 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 00:10:24.019646 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586195 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586198 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586201 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586203 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586206 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586210 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586212 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586215 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586218 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586220 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586223 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586226 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586228 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586231 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586233 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586236 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586238 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586241 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586244 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 00:10:24.020241 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586248 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586251 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586254 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586257 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586260 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586263 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586266 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586268 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586271 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586274 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586276 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586282 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586284 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586287 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586290 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586293 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586295 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586298 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586300 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 00:10:24.020749 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586306 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586308 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586311 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586314 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586316 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586319 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586321 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586324 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586327 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586329 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586332 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586334 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586337 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586339 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586342 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586345 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586348 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586351 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586354 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586356 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 00:10:24.021298 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:23.586359 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.586364 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.586510 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.588778 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.590318 2585 server.go:1019] "Starting client certificate rotation" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.590425 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.590470 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.593136 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.594603 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.607904 2585 log.go:25] "Validated CRI v1 runtime API" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.611562 2585 log.go:25] "Validated CRI v1 image API" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.614430 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 00:10:24.021909 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.617420 2585 fs.go:135] Filesystem UUIDs: map[2b6931d8-49c4-43de-a13b-10347ac2fa35:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e27883b0-a954-4271-8d16-35776c995b5a:/dev/nvme0n1p3] Jul 24 00:10:24.022287 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.617436 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 00:10:24.022287 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.623870 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.623989 2585 manager.go:216] Machine: {Timestamp:2026-07-24 00:10:23.622646081 +0000 UTC m=+0.355281103 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100474 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2442eefdf53c05033ae914fdd3a616 SystemUUID:ec2442ee-fdf5-3c05-033a-e914fdd3a616 BootID:305decd5-26b4-4ae8-ad45-2d8ebd9bfc0e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:aa:8e:85:91:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:aa:8e:85:91:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:09:40:eb:a9:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.624107 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.624215 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.627126 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.627162 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.627882 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.627898 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.627913 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.628490 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.629334 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631172 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631308 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631337 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631347 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631363 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631374 2585 kubelet.go:398] "Adding apiserver pod source" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631383 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 00:10:24.022706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631753 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.631772 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.634200 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.636250 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.636278 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637871 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637892 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637901 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637906 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637915 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637921 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637929 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637934 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637943 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637951 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637964 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.637970 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.638730 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.638741 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.642108 2585 server.go:1268] "Started kubelet" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.642194 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.642291 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.642367 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.643827 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.643964 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.643975 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.645093 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.645779 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.649345 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.652962 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.652979 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.653062 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.652547 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-102.ec2.internal.18c51146ece4d72d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-102.ec2.internal,UID:ip-10-0-136-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-102.ec2.internal,},FirstTimestamp:2026-07-24 00:10:23.642072877 +0000 UTC m=+0.374707965,LastTimestamp:2026-07-24 00:10:23.642072877 +0000 UTC m=+0.374707965,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-102.ec2.internal,}" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.653601 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.653603 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.653647 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.653820 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.653831 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654462 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654476 2585 factory.go:55] Registering systemd factory Jul 24 00:10:24.149427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654485 2585 factory.go:223] Registration of the systemd container factory successfully Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654835 2585 factory.go:153] Registering CRI-O factory Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654848 2585 factory.go:223] Registration of the crio container factory successfully Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654884 2585 factory.go:103] Registering Raw factory Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.654898 2585 manager.go:1196] Started watching for new ooms in manager Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.655297 2585 manager.go:349] Starting recovery of all containers Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.657323 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.657458 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.666967 2585 manager.go:354] Recovery completed Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.671510 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.674036 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.674085 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.674100 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.676839 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.676850 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.676853 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.676931 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.676900 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-102.ec2.internal.18c51146eecd0b69 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-102.ec2.internal,UID:ip-10-0-136-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-102.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-102.ec2.internal,},FirstTimestamp:2026-07-24 00:10:23.674067817 +0000 UTC m=+0.406702837,LastTimestamp:2026-07-24 00:10:23.674067817 +0000 UTC m=+0.406702837,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-102.ec2.internal,}" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.677728 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.677760 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.677771 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.678765 2585 policy_none.go:49] "None policy: Start" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.678778 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.678790 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.679804 2585 policy_none.go:47] "Start" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.680546 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lb295" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.687546 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lb295" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.689275 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-102.ec2.internal.18c51146eecd6dde default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-102.ec2.internal,UID:ip-10-0-136-102.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-102.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-102.ec2.internal,},FirstTimestamp:2026-07-24 00:10:23.674093022 +0000 UTC m=+0.406728044,LastTimestamp:2026-07-24 00:10:23.674093022 +0000 UTC m=+0.406728044,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-102.ec2.internal,}" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.754102 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.791367 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.792607 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.792647 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.792671 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 00:10:24.150762 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.792710 2585 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.794692 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.854152 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.861920 2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-102.ec2.internal\" not found" node="ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.875694 2585 manager.go:337] "Starting Device Plugin manager" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.875767 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.875778 2585 server.go:87] "Starting device plugin registration server" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.876046 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.876058 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.876160 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.876246 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.876254 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.876772 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.876821 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.892886 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal"] Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.893101 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.894706 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.894735 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.894745 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.895996 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896138 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896174 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896746 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896756 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896772 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896777 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896782 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.896788 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.897929 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.897949 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.898634 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.898659 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.898668 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.924779 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-102.ec2.internal\" not found" node="ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.928443 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-102.ec2.internal\" not found" node="ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.954950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.152605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.954976 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d1c0db1c0eb2e7cc88fd5e78b49f45-config\") pod \"kube-apiserver-proxy-ip-10-0-136-102.ec2.internal\" (UID: \"76d1c0db1c0eb2e7cc88fd5e78b49f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.954994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.977122 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.978206 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientMemory" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.978231 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.978241 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeHasSufficientPID" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.978267 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:23.989372 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:23.989397 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-102.ec2.internal\": node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.020744 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.059171 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d1c0db1c0eb2e7cc88fd5e78b49f45-config\") pod \"kube-apiserver-proxy-ip-10-0-136-102.ec2.internal\" (UID: \"76d1c0db1c0eb2e7cc88fd5e78b49f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.059203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.059226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.059255 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.059265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76d1c0db1c0eb2e7cc88fd5e78b49f45-config\") pod \"kube-apiserver-proxy-ip-10-0-136-102.ec2.internal\" (UID: \"76d1c0db1c0eb2e7cc88fd5e78b49f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.059268 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8787e14cefbe48afd866da4ad18abf46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal\" (UID: \"8787e14cefbe48afd866da4ad18abf46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.153712 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.122415 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.263371 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.223058 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.263371 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.228237 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.263371 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.232076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.371647 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.323555 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.482250 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.424078 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-102.ec2.internal\" not found" Jul 24 00:10:24.486956 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.486206 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.553485 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.567518 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.574455 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.581794 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.589991 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.590198 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 00:10:24.592406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.590226 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.632082 2585 apiserver.go:52] "Watching apiserver" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.648634 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.650803 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-lv9cg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal","openshift-multus/multus-additional-cni-plugins-tdltg","openshift-network-diagnostics/network-check-target-rjqbq","openshift-ovn-kubernetes/ovnkube-node-hg7n2","kube-system/global-pull-secret-syncer-djmtj","kube-system/konnectivity-agent-lzc8x","kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal","openshift-dns/node-resolver-xjx9t","openshift-image-registry/node-ca-nvdwf","openshift-multus/multus-878m4","openshift-multus/network-metrics-daemon-thh2r","openshift-network-operator/iptables-alerter-v944d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7"] Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.652514 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.653119 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.654021 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.655317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.655315 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2jzxh\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.655438 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.655441 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.655545 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.656745 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.656810 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.658250 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659131 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659168 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659192 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659460 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659505 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659637 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxsd9\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659703 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659731 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659753 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dzd2v\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.659784 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.660024 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.660772 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.660872 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.660911 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.660946 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.661247 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.661253 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.661513 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:24.708403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.661547 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7l755\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.661562 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d88949d-b7a2-4105-9d66-793b76308f2b-serviceca\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpck\" (UniqueName: \"kubernetes.io/projected/1d88949d-b7a2-4105-9d66-793b76308f2b-kube-api-access-twpck\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1047a0af-167f-4337-b2bf-d34f9c64ed3b-iptables-alerter-script\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d88949d-b7a2-4105-9d66-793b76308f2b-host\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-878m4" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662808 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1047a0af-167f-4337-b2bf-d34f9c64ed3b-host-slash\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.662864 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rgm\" (UniqueName: \"kubernetes.io/projected/1047a0af-167f-4337-b2bf-d34f9c64ed3b-kube-api-access-n7rgm\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.665129 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7znpr\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.665701 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.665783 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.665702 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.666128 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.666180 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.666221 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hcvp8\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.667784 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.669263 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670387 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670647 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zs9qf\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.670712 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670763 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670781 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5rq8f\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670781 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.670822 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.671794 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2r45s\"" Jul 24 00:10:24.710376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.671874 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 00:10:24.711996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.672156 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 00:10:24.711996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.672905 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 00:10:24.711996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.693208 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zj2fz" Jul 24 00:10:24.711996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.694241 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 00:05:24 +0000 UTC" deadline="2028-04-17 20:45:40.114607854 +0000 UTC" Jul 24 00:10:24.711996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.694311 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15212h35m15.420300978s" Jul 24 00:10:24.711996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.701435 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zj2fz" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.755112 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764057 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1047a0af-167f-4337-b2bf-d34f9c64ed3b-host-slash\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764117 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysctl-d\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-run-netns\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1047a0af-167f-4337-b2bf-d34f9c64ed3b-host-slash\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764186 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c1b0e6d-0613-4706-84a1-795bbef96de4-cni-binary-copy\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbe738-d480-4b22-ad50-a0a84c52db0a-konnectivity-ca\") pod \"konnectivity-agent-lzc8x\" (UID: \"8cdbe738-d480-4b22-ad50-a0a84c52db0a\") " pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764287 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpck\" (UniqueName: \"kubernetes.io/projected/1d88949d-b7a2-4105-9d66-793b76308f2b-kube-api-access-twpck\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764307 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-etc-selinux\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764324 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764349 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-cnibin\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-etc-kubernetes\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764408 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cnibin\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-systemd-units\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-slash\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-ovnkube-config\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.828858 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764529 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-env-overrides\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764559 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-multus-certs\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764610 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1047a0af-167f-4337-b2bf-d34f9c64ed3b-iptables-alerter-script\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764635 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysconfig\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee0c4f9c-792d-4b61-9493-44a83f3632ba-hosts-file\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764685 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764712 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-k8s-cni-cncf-io\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-kubelet\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764783 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgzn\" (UniqueName: \"kubernetes.io/projected/fe43aeb9-f840-4ad5-9d41-fb217acde334-kube-api-access-7tgzn\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-etc-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-socket-dir-parent\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764911 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-hostroot\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764935 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2blk\" (UniqueName: \"kubernetes.io/projected/e49f38c5-3582-4bc1-8648-9f934a16ea11-kube-api-access-z2blk\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.764990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3632857f-077a-4c1e-b064-dcb861c45095-dbus\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765043 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.830459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765072 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-kubelet\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-log-socket\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765119 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-netns\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-cni-bin\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765161 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqdf\" (UniqueName: \"kubernetes.io/projected/833f0c21-b425-49fd-a639-2e2b235f1e7c-kube-api-access-clqdf\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765185 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-var-lib-kubelet\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765205 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e49f38c5-3582-4bc1-8648-9f934a16ea11-tmp\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765229 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765250 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-system-cni-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765272 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765307 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-sys\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-var-lib-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-ovn\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765385 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-daemon-config\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rgm\" (UniqueName: \"kubernetes.io/projected/1047a0af-167f-4337-b2bf-d34f9c64ed3b-kube-api-access-n7rgm\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765433 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-kubernetes\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.831273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4f9c-792d-4b61-9493-44a83f3632ba-tmp-dir\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765554 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-system-cni-dir\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765575 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-os-release\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765598 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d88949d-b7a2-4105-9d66-793b76308f2b-serviceca\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765652 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-cni-bin\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-ovnkube-script-lib\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765758 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdff\" (UniqueName: \"kubernetes.io/projected/12884499-b99b-49f6-bf0d-1923e417177d-kube-api-access-chdff\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-conf-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765830 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8cdbe738-d480-4b22-ad50-a0a84c52db0a-agent-certs\") pod \"konnectivity-agent-lzc8x\" (UID: \"8cdbe738-d480-4b22-ad50-a0a84c52db0a\") " pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-device-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765922 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-host\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-tuned\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-os-release\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.765997 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-cni-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766017 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysctl-conf\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.832248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766048 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xp5\" (UniqueName: \"kubernetes.io/projected/ee0c4f9c-792d-4b61-9493-44a83f3632ba-kube-api-access-d9xp5\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3632857f-077a-4c1e-b064-dcb861c45095-kubelet-config\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766150 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-node-log\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-cni-netd\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-cni-multus\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766244 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766260 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-lib-modules\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-systemd\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d88949d-b7a2-4105-9d66-793b76308f2b-host\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766341 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-socket-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766369 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-registration-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d88949d-b7a2-4105-9d66-793b76308f2b-host\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766394 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-sys-fs\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766436 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-modprobe-d\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-systemd\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.833107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-run\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766517 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766542 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nh4r\" (UniqueName: \"kubernetes.io/projected/acb33309-ccf8-4df1-a74a-2f53fc79bee3-kube-api-access-6nh4r\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766567 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766590 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12884499-b99b-49f6-bf0d-1923e417177d-ovn-node-metrics-cert\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.766624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wpm\" (UniqueName: \"kubernetes.io/projected/0c1b0e6d-0613-4706-84a1-795bbef96de4-kube-api-access-p5wpm\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.771526 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1047a0af-167f-4337-b2bf-d34f9c64ed3b-iptables-alerter-script\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.773617 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d88949d-b7a2-4105-9d66-793b76308f2b-serviceca\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.774908 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.778436 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rgm\" (UniqueName: \"kubernetes.io/projected/1047a0af-167f-4337-b2bf-d34f9c64ed3b-kube-api-access-n7rgm\") pod \"iptables-alerter-v944d\" (UID: \"1047a0af-167f-4337-b2bf-d34f9c64ed3b\") " pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:24.836028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.778473 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpck\" (UniqueName: \"kubernetes.io/projected/1d88949d-b7a2-4105-9d66-793b76308f2b-kube-api-access-twpck\") pod \"node-ca-nvdwf\" (UID: \"1d88949d-b7a2-4105-9d66-793b76308f2b\") " pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.866942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-run\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.866995 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867022 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nh4r\" (UniqueName: \"kubernetes.io/projected/acb33309-ccf8-4df1-a74a-2f53fc79bee3-kube-api-access-6nh4r\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867068 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12884499-b99b-49f6-bf0d-1923e417177d-ovn-node-metrics-cert\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867093 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wpm\" (UniqueName: \"kubernetes.io/projected/0c1b0e6d-0613-4706-84a1-795bbef96de4-kube-api-access-p5wpm\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867113 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-run\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysctl-d\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867201 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-run-netns\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c1b0e6d-0613-4706-84a1-795bbef96de4-cni-binary-copy\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867259 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbe738-d480-4b22-ad50-a0a84c52db0a-konnectivity-ca\") pod \"konnectivity-agent-lzc8x\" (UID: \"8cdbe738-d480-4b22-ad50-a0a84c52db0a\") " pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-run-netns\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-etc-selinux\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.867232 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867326 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-cnibin\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867418 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-cnibin\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867421 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-etc-selinux\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.867427 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:25.367378134 +0000 UTC m=+2.100013157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-etc-kubernetes\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867487 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cnibin\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-systemd-units\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867548 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-etc-kubernetes\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867561 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cnibin\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867572 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-systemd-units\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867599 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-slash\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-ovnkube-config\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867637 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-env-overrides\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-multus-certs\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867682 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-slash\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867705 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysctl-d\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867706 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-multus-certs\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867739 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysconfig\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.952795 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee0c4f9c-792d-4b61-9493-44a83f3632ba-hosts-file\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-k8s-cni-cncf-io\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867843 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ee0c4f9c-792d-4b61-9493-44a83f3632ba-hosts-file\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867845 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-k8s-cni-cncf-io\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867845 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysconfig\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867875 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-kubelet\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867856 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867903 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgzn\" (UniqueName: \"kubernetes.io/projected/fe43aeb9-f840-4ad5-9d41-fb217acde334-kube-api-access-7tgzn\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867913 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-kubelet\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.867922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868036 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-etc-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868060 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-socket-dir-parent\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-hostroot\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868101 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2blk\" (UniqueName: \"kubernetes.io/projected/e49f38c5-3582-4bc1-8648-9f934a16ea11-kube-api-access-z2blk\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868122 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3632857f-077a-4c1e-b064-dcb861c45095-dbus\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868151 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.955912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-hostroot\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-kubelet\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868192 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-log-socket\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868147 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-socket-dir-parent\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868236 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-netns\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868263 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-etc-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868267 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-cni-bin\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868306 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqdf\" (UniqueName: \"kubernetes.io/projected/833f0c21-b425-49fd-a639-2e2b235f1e7c-kube-api-access-clqdf\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868237 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-log-socket\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868319 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3632857f-077a-4c1e-b064-dcb861c45095-dbus\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-var-lib-kubelet\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868332 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-run-netns\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868301 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-cni-bin\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868356 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-kubelet\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868371 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e49f38c5-3582-4bc1-8648-9f934a16ea11-tmp\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868397 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868419 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-system-cni-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868423 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-var-lib-kubelet\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.956677 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-sys\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868476 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868488 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-var-lib-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868509 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-ovn\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868531 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-daemon-config\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868558 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-kubernetes\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868569 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-sys\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868598 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-ovn\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868609 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-kubernetes\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868619 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-system-cni-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868636 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4f9c-792d-4b61-9493-44a83f3632ba-tmp-dir\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868661 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868669 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-system-cni-dir\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-os-release\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.959409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-cni-bin\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-system-cni-dir\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868767 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-ovnkube-script-lib\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868662 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-var-lib-openvswitch\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868798 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-os-release\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-cni-bin\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868882 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdff\" (UniqueName: \"kubernetes.io/projected/12884499-b99b-49f6-bf0d-1923e417177d-kube-api-access-chdff\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-conf-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868927 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8cdbe738-d480-4b22-ad50-a0a84c52db0a-agent-certs\") pod \"konnectivity-agent-lzc8x\" (UID: \"8cdbe738-d480-4b22-ad50-a0a84c52db0a\") " pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-device-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-host\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868984 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-tuned\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.868990 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-conf-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869002 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-os-release\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-cni-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869039 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysctl-conf\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xp5\" (UniqueName: \"kubernetes.io/projected/ee0c4f9c-792d-4b61-9493-44a83f3632ba-kube-api-access-d9xp5\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:24.960190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869072 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3632857f-077a-4c1e-b064-dcb861c45095-kubelet-config\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869089 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869105 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-node-log\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-cni-netd\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869137 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-cni-multus\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-lib-modules\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-systemd\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-socket-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869233 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-registration-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-sys-fs\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869267 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-modprobe-d\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869286 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-systemd\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869307 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-sysctl-conf\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869362 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3632857f-077a-4c1e-b064-dcb861c45095-kubelet-config\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869625 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-lib-modules\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869671 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-node-log\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.964050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869709 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-run-systemd\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869715 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12884499-b99b-49f6-bf0d-1923e417177d-host-cni-netd\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869766 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-host-var-lib-cni-multus\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869836 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-socket-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.869846 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-registration-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-cni-dir\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.869914 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:10:25.369895441 +0000 UTC m=+2.102530462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869979 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-device-dir\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-host\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.869996 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-modprobe-d\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.870031 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-systemd\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.870033 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fe43aeb9-f840-4ad5-9d41-fb217acde334-sys-fs\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.870124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acb33309-ccf8-4df1-a74a-2f53fc79bee3-os-release\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.880572 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.880595 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.880607 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:24.964805 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:24.880678 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:25.380660392 +0000 UTC m=+2.113295397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.992745 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c1b0e6d-0613-4706-84a1-795bbef96de4-cni-binary-copy\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.992834 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c1b0e6d-0613-4706-84a1-795bbef96de4-multus-daemon-config\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993088 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-ovnkube-config\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993223 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993388 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-env-overrides\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993461 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbe738-d480-4b22-ad50-a0a84c52db0a-konnectivity-ca\") pod \"konnectivity-agent-lzc8x\" (UID: \"8cdbe738-d480-4b22-ad50-a0a84c52db0a\") " pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993499 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee0c4f9c-792d-4b61-9493-44a83f3632ba-tmp-dir\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993738 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993780 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12884499-b99b-49f6-bf0d-1923e417177d-ovnkube-script-lib\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.993935 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acb33309-ccf8-4df1-a74a-2f53fc79bee3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.996456 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12884499-b99b-49f6-bf0d-1923e417177d-ovn-node-metrics-cert\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.996561 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e49f38c5-3582-4bc1-8648-9f934a16ea11-tmp\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.997005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgzn\" (UniqueName: \"kubernetes.io/projected/fe43aeb9-f840-4ad5-9d41-fb217acde334-kube-api-access-7tgzn\") pod \"aws-ebs-csi-driver-node-pqqx7\" (UID: \"fe43aeb9-f840-4ad5-9d41-fb217acde334\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.997834 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xp5\" (UniqueName: \"kubernetes.io/projected/ee0c4f9c-792d-4b61-9493-44a83f3632ba-kube-api-access-d9xp5\") pod \"node-resolver-xjx9t\" (UID: \"ee0c4f9c-792d-4b61-9493-44a83f3632ba\") " pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.998265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2blk\" (UniqueName: \"kubernetes.io/projected/e49f38c5-3582-4bc1-8648-9f934a16ea11-kube-api-access-z2blk\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.998365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e49f38c5-3582-4bc1-8648-9f934a16ea11-etc-tuned\") pod \"tuned-lv9cg\" (UID: \"e49f38c5-3582-4bc1-8648-9f934a16ea11\") " pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:25.080050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.998459 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqdf\" (UniqueName: \"kubernetes.io/projected/833f0c21-b425-49fd-a639-2e2b235f1e7c-kube-api-access-clqdf\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:25.083160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.998578 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wpm\" (UniqueName: \"kubernetes.io/projected/0c1b0e6d-0613-4706-84a1-795bbef96de4-kube-api-access-p5wpm\") pod \"multus-878m4\" (UID: \"0c1b0e6d-0613-4706-84a1-795bbef96de4\") " pod="openshift-multus/multus-878m4" Jul 24 00:10:25.083160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.998656 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdff\" (UniqueName: \"kubernetes.io/projected/12884499-b99b-49f6-bf0d-1923e417177d-kube-api-access-chdff\") pod \"ovnkube-node-hg7n2\" (UID: \"12884499-b99b-49f6-bf0d-1923e417177d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:25.083160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.998672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nh4r\" (UniqueName: \"kubernetes.io/projected/acb33309-ccf8-4df1-a74a-2f53fc79bee3-kube-api-access-6nh4r\") pod \"multus-additional-cni-plugins-tdltg\" (UID: \"acb33309-ccf8-4df1-a74a-2f53fc79bee3\") " pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:25.083160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:24.999782 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8cdbe738-d480-4b22-ad50-a0a84c52db0a-agent-certs\") pod \"konnectivity-agent-lzc8x\" (UID: \"8cdbe738-d480-4b22-ad50-a0a84c52db0a\") " pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:25.197235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.151981 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:10:25.197235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.171595 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:10:25.373702 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.373582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:25.373702 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.373649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:25.373953 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.373749 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:25.373953 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.373778 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:25.373953 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.373831 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:10:26.373809212 +0000 UTC m=+3.106444233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:25.373953 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.373856 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:26.373849129 +0000 UTC m=+3.106484132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:25.453076 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.453039 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nvdwf" Jul 24 00:10:25.463459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.463436 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v944d" Jul 24 00:10:25.474367 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.474336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:25.474556 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.474517 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:25.474633 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.474559 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:25.474633 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.474570 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:25.474752 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.474634 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:26.474619372 +0000 UTC m=+3.207254391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:25.478646 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.478623 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tdltg" Jul 24 00:10:25.494007 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.493969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:25.516293 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.516257 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" Jul 24 00:10:25.537215 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.537188 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-878m4" Jul 24 00:10:25.552140 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.552109 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" Jul 24 00:10:25.572819 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.572792 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:25.578639 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.578600 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49f38c5_3582_4bc1_8648_9f934a16ea11.slice/crio-1cb8b7c2fd58cfe326b113a87d9c9df4a363701cbffe977b2d37caef3893ff69 WatchSource:0}: Error finding container 1cb8b7c2fd58cfe326b113a87d9c9df4a363701cbffe977b2d37caef3893ff69: Status 404 returned error can't find the container with id 1cb8b7c2fd58cfe326b113a87d9c9df4a363701cbffe977b2d37caef3893ff69 Jul 24 00:10:25.578925 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.578904 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d1c0db1c0eb2e7cc88fd5e78b49f45.slice/crio-c1a22449290bde52f6535e66403b69149d19afcf3f50469b508e711d805966a2 WatchSource:0}: Error finding container c1a22449290bde52f6535e66403b69149d19afcf3f50469b508e711d805966a2: Status 404 returned error can't find the container with id c1a22449290bde52f6535e66403b69149d19afcf3f50469b508e711d805966a2 Jul 24 00:10:25.586303 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.586282 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:10:25.587309 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.587286 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdbe738_d480_4b22_ad50_a0a84c52db0a.slice/crio-e57c032ae4e7d85e01b5808ff196b4a1a06ea83be70305ad7d5f348ad4544fb4 WatchSource:0}: Error finding container e57c032ae4e7d85e01b5808ff196b4a1a06ea83be70305ad7d5f348ad4544fb4: Status 404 returned error can't find the container with id e57c032ae4e7d85e01b5808ff196b4a1a06ea83be70305ad7d5f348ad4544fb4 Jul 24 00:10:25.587426 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.587407 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xjx9t" Jul 24 00:10:25.596221 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.596195 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0c4f9c_792d_4b61_9493_44a83f3632ba.slice/crio-71361824da58d8d81ef7e84f75318178da86a9be0f82727b55ab761ae94dbbed WatchSource:0}: Error finding container 71361824da58d8d81ef7e84f75318178da86a9be0f82727b55ab761ae94dbbed: Status 404 returned error can't find the container with id 71361824da58d8d81ef7e84f75318178da86a9be0f82727b55ab761ae94dbbed Jul 24 00:10:25.704424 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.704292 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:05:25 +0000 UTC" deadline="2028-03-07 14:34:40.417550064 +0000 UTC" Jul 24 00:10:25.704424 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.704354 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14222h24m14.713198846s" Jul 24 00:10:25.762977 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.762943 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1047a0af_167f_4337_b2bf_d34f9c64ed3b.slice/crio-b6626d3925889f864be18724343f710ae534a6d88d4dc2d42e7913a163575035 WatchSource:0}: Error finding container b6626d3925889f864be18724343f710ae534a6d88d4dc2d42e7913a163575035: Status 404 returned error can't find the container with id b6626d3925889f864be18724343f710ae534a6d88d4dc2d42e7913a163575035 Jul 24 00:10:25.796806 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.796777 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:25.797021 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:25.796892 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:25.797666 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.797624 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" event={"ID":"76d1c0db1c0eb2e7cc88fd5e78b49f45","Type":"ContainerStarted","Data":"c1a22449290bde52f6535e66403b69149d19afcf3f50469b508e711d805966a2"} Jul 24 00:10:25.798834 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.798813 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" event={"ID":"e49f38c5-3582-4bc1-8648-9f934a16ea11","Type":"ContainerStarted","Data":"1cb8b7c2fd58cfe326b113a87d9c9df4a363701cbffe977b2d37caef3893ff69"} Jul 24 00:10:25.800128 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.800103 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v944d" event={"ID":"1047a0af-167f-4337-b2bf-d34f9c64ed3b","Type":"ContainerStarted","Data":"b6626d3925889f864be18724343f710ae534a6d88d4dc2d42e7913a163575035"} Jul 24 00:10:25.801158 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.801128 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjx9t" event={"ID":"ee0c4f9c-792d-4b61-9493-44a83f3632ba","Type":"ContainerStarted","Data":"71361824da58d8d81ef7e84f75318178da86a9be0f82727b55ab761ae94dbbed"} Jul 24 00:10:25.802247 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:25.802211 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzc8x" event={"ID":"8cdbe738-d480-4b22-ad50-a0a84c52db0a","Type":"ContainerStarted","Data":"e57c032ae4e7d85e01b5808ff196b4a1a06ea83be70305ad7d5f348ad4544fb4"} Jul 24 00:10:25.843308 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.843278 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d88949d_b7a2_4105_9d66_793b76308f2b.slice/crio-a0563aa35fafcf95e6c4e04f58bfe766404e66bb58432b6ce0b7e29176c988ec WatchSource:0}: Error finding container a0563aa35fafcf95e6c4e04f58bfe766404e66bb58432b6ce0b7e29176c988ec: Status 404 returned error can't find the container with id a0563aa35fafcf95e6c4e04f58bfe766404e66bb58432b6ce0b7e29176c988ec Jul 24 00:10:25.860191 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.860014 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12884499_b99b_49f6_bf0d_1923e417177d.slice/crio-1126db9a275a36cea1075816d1e41d3424bd729e5a1486f5200eec02e1c942e6 WatchSource:0}: Error finding container 1126db9a275a36cea1075816d1e41d3424bd729e5a1486f5200eec02e1c942e6: Status 404 returned error can't find the container with id 1126db9a275a36cea1075816d1e41d3424bd729e5a1486f5200eec02e1c942e6 Jul 24 00:10:25.941759 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:25.941717 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe43aeb9_f840_4ad5_9d41_fb217acde334.slice/crio-b8ae1d37f35bf2316b71a7b3b135d3474d2ae2aad2c0e4d27d471aa3919a3524 WatchSource:0}: Error finding container b8ae1d37f35bf2316b71a7b3b135d3474d2ae2aad2c0e4d27d471aa3919a3524: Status 404 returned error can't find the container with id b8ae1d37f35bf2316b71a7b3b135d3474d2ae2aad2c0e4d27d471aa3919a3524 Jul 24 00:10:26.027586 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:26.027540 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb33309_ccf8_4df1_a74a_2f53fc79bee3.slice/crio-2c5edba9438d9e5791e0323dbc2da2c2ed76852400e8715676e28793c46bc27d WatchSource:0}: Error finding container 2c5edba9438d9e5791e0323dbc2da2c2ed76852400e8715676e28793c46bc27d: Status 404 returned error can't find the container with id 2c5edba9438d9e5791e0323dbc2da2c2ed76852400e8715676e28793c46bc27d Jul 24 00:10:26.030537 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:26.030504 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1b0e6d_0613_4706_84a1_795bbef96de4.slice/crio-3bb74cd74f13d23c72dd7fd12ed38b2f3e0acc1a04ead7344eb12a133381cf53 WatchSource:0}: Error finding container 3bb74cd74f13d23c72dd7fd12ed38b2f3e0acc1a04ead7344eb12a133381cf53: Status 404 returned error can't find the container with id 3bb74cd74f13d23c72dd7fd12ed38b2f3e0acc1a04ead7344eb12a133381cf53 Jul 24 00:10:26.059833 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.059801 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:10:26.073284 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:10:26.073245 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8787e14cefbe48afd866da4ad18abf46.slice/crio-abecbcf4b809fe9355efdbd7800bf3502d6669a063c7d932ed062fc10eeaeecf WatchSource:0}: Error finding container abecbcf4b809fe9355efdbd7800bf3502d6669a063c7d932ed062fc10eeaeecf: Status 404 returned error can't find the container with id abecbcf4b809fe9355efdbd7800bf3502d6669a063c7d932ed062fc10eeaeecf Jul 24 00:10:26.384506 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.383947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:26.384506 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.384009 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:26.384506 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.384193 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:26.384506 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.384230 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:26.384506 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.384262 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:28.384239252 +0000 UTC m=+5.116874262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:26.384506 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.384317 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:10:28.384298884 +0000 UTC m=+5.116933890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:26.484929 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.484873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:26.485149 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.485093 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:26.485149 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.485135 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:26.485149 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.485148 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:26.485330 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.485216 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:28.485193181 +0000 UTC m=+5.217828202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:26.705235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.705122 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 00:05:25 +0000 UTC" deadline="2028-02-02 10:54:45.422012409 +0000 UTC" Jul 24 00:10:26.705235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.705170 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13402h44m18.716847465s" Jul 24 00:10:26.794389 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.793860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:26.794389 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.794011 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:26.794389 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.794034 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:26.794389 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:26.794162 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:26.808843 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.808799 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" event={"ID":"8787e14cefbe48afd866da4ad18abf46","Type":"ContainerStarted","Data":"abecbcf4b809fe9355efdbd7800bf3502d6669a063c7d932ed062fc10eeaeecf"} Jul 24 00:10:26.817808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.817638 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" event={"ID":"fe43aeb9-f840-4ad5-9d41-fb217acde334","Type":"ContainerStarted","Data":"b8ae1d37f35bf2316b71a7b3b135d3474d2ae2aad2c0e4d27d471aa3919a3524"} Jul 24 00:10:26.843700 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.843627 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"1126db9a275a36cea1075816d1e41d3424bd729e5a1486f5200eec02e1c942e6"} Jul 24 00:10:26.864299 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.864255 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nvdwf" event={"ID":"1d88949d-b7a2-4105-9d66-793b76308f2b","Type":"ContainerStarted","Data":"a0563aa35fafcf95e6c4e04f58bfe766404e66bb58432b6ce0b7e29176c988ec"} Jul 24 00:10:26.890253 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.890209 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-878m4" event={"ID":"0c1b0e6d-0613-4706-84a1-795bbef96de4","Type":"ContainerStarted","Data":"3bb74cd74f13d23c72dd7fd12ed38b2f3e0acc1a04ead7344eb12a133381cf53"} Jul 24 00:10:26.895337 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:26.894962 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerStarted","Data":"2c5edba9438d9e5791e0323dbc2da2c2ed76852400e8715676e28793c46bc27d"} Jul 24 00:10:27.213511 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:27.213474 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 00:10:27.796217 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:27.796165 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:27.796883 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:27.796324 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:28.407628 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:28.407585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:28.407853 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:28.407649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:28.407853 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.407814 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:28.408026 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.407885 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:32.407864167 +0000 UTC m=+9.140499193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:28.408391 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.408366 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:28.408485 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.408433 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:10:32.408414798 +0000 UTC m=+9.141049821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:28.508619 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:28.508573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:28.508841 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.508821 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:28.508917 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.508848 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:28.508917 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.508862 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:28.509058 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.508941 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:32.508920842 +0000 UTC m=+9.241555871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:28.794189 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:28.793551 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:28.794189 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:28.793593 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:28.794189 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.793705 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:28.794189 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:28.793847 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:29.794699 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:29.794149 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:29.794699 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:29.794306 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:30.793190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:30.793158 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:30.793417 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:30.793284 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:30.793417 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:30.793159 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:30.793566 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:30.793424 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:31.793999 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:31.793461 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:31.793999 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:31.793603 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:32.445390 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:32.445336 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:32.445605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:32.445462 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:32.445605 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.445599 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:32.445702 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.445665 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:10:40.445650301 +0000 UTC m=+17.178285306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:32.446118 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.446010 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:32.446118 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.446092 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:40.446071557 +0000 UTC m=+17.178706562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:32.546520 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:32.546468 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:32.546738 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.546674 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:32.546738 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.546701 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:32.546738 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.546716 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:32.546933 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.546781 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:40.546759434 +0000 UTC m=+17.279394458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:32.793676 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:32.793090 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:32.793676 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.793233 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:32.793676 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:32.793272 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:32.793676 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:32.793415 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:33.794708 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:33.794673 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:33.795181 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:33.794770 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:34.793579 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:34.793539 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:34.793775 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:34.793681 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:34.794066 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:34.793539 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:34.794198 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:34.794160 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:35.793930 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:35.793828 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:35.794359 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:35.793993 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:36.793620 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:36.793107 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:36.793620 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:36.793262 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:36.793620 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:36.793288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:36.793620 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:36.793423 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:37.793528 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:37.793484 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:37.794042 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:37.793628 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:38.793958 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:38.793877 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:38.793958 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:38.793877 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:38.794438 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:38.794029 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:38.794438 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:38.794173 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:39.793313 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:39.793276 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:39.793514 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:39.793391 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:40.514305 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:40.514255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:40.514745 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:40.514321 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:40.514745 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.514411 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:40.514745 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.514463 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:10:56.514448878 +0000 UTC m=+33.247083884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:40.514745 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.514480 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:40.514745 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.514578 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:56.514559486 +0000 UTC m=+33.247194492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:40.615694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:40.615645 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:40.615885 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.615843 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:40.615885 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.615878 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:40.615997 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.615892 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:40.615997 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.615960 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:10:56.615942027 +0000 UTC m=+33.348577031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:40.793789 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:40.793693 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:40.793789 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:40.793732 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:40.793987 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.793832 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:40.794040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:40.794011 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:41.793827 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:41.793787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:41.794230 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:41.793941 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:42.793257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:42.793169 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:42.793257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:42.793200 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:42.793476 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:42.793313 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:42.793531 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:42.793466 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:43.794710 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:43.794675 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:43.795163 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:43.794797 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:44.793327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:44.793297 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:44.793523 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:44.793299 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:44.793523 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:44.793429 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:44.793523 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:44.793497 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:45.793678 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:45.793643 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:45.794155 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:45.793776 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:46.793548 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.793168 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:46.793548 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:46.793520 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:46.794394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.794366 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:46.794515 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:46.794469 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:46.946541 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.946267 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" event={"ID":"76d1c0db1c0eb2e7cc88fd5e78b49f45","Type":"ContainerStarted","Data":"01c8fc4f9015a740c06e4a4e443a2d1683565c889707dd31597ff18f4a57830c"} Jul 24 00:10:46.954606 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.953969 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" event={"ID":"e49f38c5-3582-4bc1-8648-9f934a16ea11","Type":"ContainerStarted","Data":"44348c49273d9351e1ed06f1cb285ad039105ea8861b21e9364b12aeaba22644"} Jul 24 00:10:46.958398 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.958364 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" event={"ID":"fe43aeb9-f840-4ad5-9d41-fb217acde334","Type":"ContainerStarted","Data":"44ab1cbd5f5f45803f4da67521a4e416254cbf5418f51ada9d4bdb5f7e14238f"} Jul 24 00:10:46.962462 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.962213 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"ed79665f110d6ba7717b62bf63a036eb54b7703e8d383b70e52d2d8c6418a55d"} Jul 24 00:10:46.962462 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.962309 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"b74fc168916d4070b380126f460d4a93d82a316b6ca8b1c2fb9306b6e33113c9"} Jul 24 00:10:46.963226 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.963111 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-102.ec2.internal" podStartSLOduration=21.963095099 podStartE2EDuration="21.963095099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:46.962706489 +0000 UTC m=+23.695341515" watchObservedRunningTime="2026-07-24 00:10:46.963095099 +0000 UTC m=+23.695730105" Jul 24 00:10:46.969234 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.969207 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzc8x" event={"ID":"8cdbe738-d480-4b22-ad50-a0a84c52db0a","Type":"ContainerStarted","Data":"41c72a0de5f1da00310880bca48c2a5d2a0318fbd2906d3aa74247c5fbc2b5c9"} Jul 24 00:10:46.972170 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.971986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-878m4" event={"ID":"0c1b0e6d-0613-4706-84a1-795bbef96de4","Type":"ContainerStarted","Data":"d04e09556d02b7ec67850979c5bb400bb86854bdec8237c42217e1b788d66492"} Jul 24 00:10:46.983733 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:46.983677 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lv9cg" podStartSLOduration=0.984783384 podStartE2EDuration="21.983654411s" totalImagesPullingTime="20.998871027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:46.982610239 +0000 UTC m=+23.715245265" watchObservedRunningTime="2026-07-24 00:10:46.983654411 +0000 UTC m=+23.716289436" Jul 24 00:10:47.000755 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.000702 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-878m4" podStartSLOduration=2.39430044 podStartE2EDuration="23.000680937s" totalImagesPullingTime="20.606380497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:47.000627091 +0000 UTC m=+23.733262188" watchObservedRunningTime="2026-07-24 00:10:47.000680937 +0000 UTC m=+23.733315964" Jul 24 00:10:47.026551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.026510 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lzc8x" podStartSLOduration=5.281461967 podStartE2EDuration="22.02649297s" totalImagesPullingTime="16.745031003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:47.025838253 +0000 UTC m=+23.758473277" watchObservedRunningTime="2026-07-24 00:10:47.02649297 +0000 UTC m=+23.759127995" Jul 24 00:10:47.797803 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.797696 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:47.797803 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:47.797796 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:47.977739 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.977564 2585 generic.go:356] "Generic (PLEG): container finished" podID="acb33309-ccf8-4df1-a74a-2f53fc79bee3" containerID="c95f17f828d93d967e95d6208183bd1befcbb055137eb52fee2a510871021092" exitCode=0 Jul 24 00:10:47.977887 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.977646 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerDied","Data":"c95f17f828d93d967e95d6208183bd1befcbb055137eb52fee2a510871021092"} Jul 24 00:10:47.979243 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.979216 2585 generic.go:356] "Generic (PLEG): container finished" podID="8787e14cefbe48afd866da4ad18abf46" containerID="78949bc60580ed869d2cd2eeb1bb635b65eac4773a070fde556b232ac41f273a" exitCode=0 Jul 24 00:10:47.979333 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.979306 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" event={"ID":"8787e14cefbe48afd866da4ad18abf46","Type":"ContainerDied","Data":"78949bc60580ed869d2cd2eeb1bb635b65eac4773a070fde556b232ac41f273a"} Jul 24 00:10:47.982236 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.982212 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"be83abc3a54376ebc0bcd25def76361b4d3770edb69fe75afd1135d14004c83f"} Jul 24 00:10:47.982323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.982279 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"473089e0c20dd1fb3a3911c0d75bb0f55fe2b16244b7d6d06a2f2bba53141caf"} Jul 24 00:10:47.982362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.982323 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"6a220791e0ad23dc13438d147db8f152072facbfba5b75b8b80e93d990c26a3f"} Jul 24 00:10:47.982362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.982357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"6f365924f5438304d12b199023d1177d2086d3017f9e218e39fecf91ba5bf436"} Jul 24 00:10:47.983503 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.983474 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nvdwf" event={"ID":"1d88949d-b7a2-4105-9d66-793b76308f2b","Type":"ContainerStarted","Data":"d5a38744e7ca4943632fc15905c515b1b9e713b03f159c1b2d85a5c27ee2f0c1"} Jul 24 00:10:47.984917 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:47.984899 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xjx9t" event={"ID":"ee0c4f9c-792d-4b61-9493-44a83f3632ba","Type":"ContainerStarted","Data":"624cab581ea7f9393bcc6105266c1b7ea684fa360c7a6c84c0368d097b7bdb51"} Jul 24 00:10:48.040191 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.040158 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:48.040646 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.040627 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:48.113655 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.113610 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xjx9t" podStartSLOduration=2.485472899 podStartE2EDuration="23.113595865s" totalImagesPullingTime="20.628122966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:48.113096119 +0000 UTC m=+24.845731145" watchObservedRunningTime="2026-07-24 00:10:48.113595865 +0000 UTC m=+24.846230887" Jul 24 00:10:48.113787 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.113683 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nvdwf" podStartSLOduration=3.397045213 podStartE2EDuration="24.113678339s" totalImagesPullingTime="20.716633126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:48.098092944 +0000 UTC m=+24.830727968" watchObservedRunningTime="2026-07-24 00:10:48.113678339 +0000 UTC m=+24.846313369" Jul 24 00:10:48.370784 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.370756 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 00:10:48.793285 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.793211 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:48.793285 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.793246 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:48.793518 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:48.793318 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:48.793518 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:48.793435 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:48.891806 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.891703 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T00:10:48.370776844Z","UUID":"5e01f321-960e-4010-94df-ea263eb1cf38","Handler":null,"Name":"","Endpoint":""} Jul 24 00:10:48.893628 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.893605 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 00:10:48.893755 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.893652 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 00:10:48.989128 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.989089 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" event={"ID":"8787e14cefbe48afd866da4ad18abf46","Type":"ContainerStarted","Data":"58c19bb1bac01c308d87aa1cd2c7e8fc2ff00da9e1d887dec43bce06606ef382"} Jul 24 00:10:48.991203 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.991170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" event={"ID":"fe43aeb9-f840-4ad5-9d41-fb217acde334","Type":"ContainerStarted","Data":"84aee75d7c53dbbab66a0940103f678d872a5ea735812650f8b4e99312d10d07"} Jul 24 00:10:48.993036 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.993002 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v944d" event={"ID":"1047a0af-167f-4337-b2bf-d34f9c64ed3b","Type":"ContainerStarted","Data":"7e9a95329452dc0170b37845cd5217408869114ae402f556a66ed002d0d3df36"} Jul 24 00:10:48.993605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.993564 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:48.994089 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:48.994070 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lzc8x" Jul 24 00:10:49.008316 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:49.008258 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-102.ec2.internal" podStartSLOduration=24.008241205 podStartE2EDuration="24.008241205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:49.007906134 +0000 UTC m=+25.740541159" watchObservedRunningTime="2026-07-24 00:10:49.008241205 +0000 UTC m=+25.740876230" Jul 24 00:10:49.024647 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:49.024588 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v944d" podStartSLOduration=4.205232515 podStartE2EDuration="25.024573022s" totalImagesPullingTime="20.819340507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:49.024180901 +0000 UTC m=+25.756815927" watchObservedRunningTime="2026-07-24 00:10:49.024573022 +0000 UTC m=+25.757208046" Jul 24 00:10:49.793476 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:49.793396 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:49.793726 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:49.793552 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:49.997177 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:49.997136 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" event={"ID":"fe43aeb9-f840-4ad5-9d41-fb217acde334","Type":"ContainerStarted","Data":"26ae55b2dc5c98e151acb2243d6a56ef7d47cd33126e905c5ece805d2869c829"} Jul 24 00:10:50.000141 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:50.000108 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"ad997d910c08d6f6345afc4ad0c94e073f8d3d990062624d8efd2e039cc20e3e"} Jul 24 00:10:50.793842 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:50.793810 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:50.794073 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:50.793872 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:50.794073 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:50.793971 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:50.794163 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:50.794132 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:51.021528 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:51.021454 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pqqx7" podStartSLOduration=3.990067882 podStartE2EDuration="27.021438057s" totalImagesPullingTime="23.031370175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:51.021382536 +0000 UTC m=+27.754017561" watchObservedRunningTime="2026-07-24 00:10:51.021438057 +0000 UTC m=+27.754073072" Jul 24 00:10:51.793498 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:51.793309 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:51.793633 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:51.793592 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:52.009746 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.009661 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" event={"ID":"12884499-b99b-49f6-bf0d-1923e417177d","Type":"ContainerStarted","Data":"e2aa1963e7b4afe53e7b18f083c0fa9ce073da58bbc96544f682596c9b64a66d"} Jul 24 00:10:52.010011 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.009982 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:52.010146 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.010132 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:52.010193 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.010184 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:52.029303 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.029273 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:52.033998 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.033976 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:10:52.040876 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.040822 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" podStartSLOduration=7.264307907 podStartE2EDuration="28.040808015s" totalImagesPullingTime="20.776500108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:10:52.038871682 +0000 UTC m=+28.771506756" watchObservedRunningTime="2026-07-24 00:10:52.040808015 +0000 UTC m=+28.773443091" Jul 24 00:10:52.793680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.793592 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:52.793844 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:52.793711 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:52.793844 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:52.793778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:52.794003 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:52.793978 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:53.797034 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:53.797002 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:53.797794 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:53.797105 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:54.038243 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:54.038205 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rjqbq"] Jul 24 00:10:54.038389 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:54.038328 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:54.038446 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:54.038421 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:54.040702 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:54.040674 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-djmtj"] Jul 24 00:10:54.040842 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:54.040774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:54.040914 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:54.040889 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:54.054613 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:54.054526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-thh2r"] Jul 24 00:10:54.054758 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:54.054625 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:54.054758 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:54.054706 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:55.018273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:55.018077 2585 generic.go:356] "Generic (PLEG): container finished" podID="acb33309-ccf8-4df1-a74a-2f53fc79bee3" containerID="d72ab25287f9d4035476ef566adcfe53b72d041826c3a4e6844a10f116c80101" exitCode=0 Jul 24 00:10:55.018693 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:55.018161 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerDied","Data":"d72ab25287f9d4035476ef566adcfe53b72d041826c3a4e6844a10f116c80101"} Jul 24 00:10:55.793464 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:55.793437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:55.793580 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:55.793473 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:55.793580 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:55.793437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:55.793580 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:55.793562 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:55.793730 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:55.793670 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:55.793798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:55.793773 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:56.023350 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:56.023265 2585 generic.go:356] "Generic (PLEG): container finished" podID="acb33309-ccf8-4df1-a74a-2f53fc79bee3" containerID="5a88764912c7384107df7513883c5a8af13f46cd6fe0e97d8a48d7fd413515b4" exitCode=0 Jul 24 00:10:56.023350 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:56.023326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerDied","Data":"5a88764912c7384107df7513883c5a8af13f46cd6fe0e97d8a48d7fd413515b4"} Jul 24 00:10:56.534931 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:56.534897 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:56.535143 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:56.534940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:56.535143 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.535046 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:56.535143 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.535048 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:56.535143 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.535110 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret podName:3632857f-077a-4c1e-b064-dcb861c45095 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:28.535096936 +0000 UTC m=+65.267731939 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret") pod "global-pull-secret-syncer-djmtj" (UID: "3632857f-077a-4c1e-b064-dcb861c45095") : object "kube-system"/"original-pull-secret" not registered Jul 24 00:10:56.535143 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.535124 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs podName:833f0c21-b425-49fd-a639-2e2b235f1e7c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:28.535118498 +0000 UTC m=+65.267753501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs") pod "network-metrics-daemon-thh2r" (UID: "833f0c21-b425-49fd-a639-2e2b235f1e7c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 00:10:56.635903 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:56.635852 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:56.636077 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.635988 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 00:10:56.636077 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.636015 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 00:10:56.636077 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.636026 2585 projected.go:196] Error preparing data for projected volume kube-api-access-4cbmr for pod openshift-network-diagnostics/network-check-target-rjqbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:56.636077 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:56.636074 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr podName:fd6a6183-624e-4420-99cd-df0beaa4ae40 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:28.636061573 +0000 UTC m=+65.368696579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4cbmr" (UniqueName: "kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr") pod "network-check-target-rjqbq" (UID: "fd6a6183-624e-4420-99cd-df0beaa4ae40") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 00:10:57.028466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:57.028435 2585 generic.go:356] "Generic (PLEG): container finished" podID="acb33309-ccf8-4df1-a74a-2f53fc79bee3" containerID="dd95a8d75faab94e23ef4b177897aae2f4bd4f5140ff66360f0d339a8a93a566" exitCode=0 Jul 24 00:10:57.028820 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:57.028496 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerDied","Data":"dd95a8d75faab94e23ef4b177897aae2f4bd4f5140ff66360f0d339a8a93a566"} Jul 24 00:10:57.793032 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:57.792998 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:57.793216 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:57.792998 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:57.793216 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:57.793127 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:10:57.793216 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:57.792998 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:57.793376 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:57.793208 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:57.793376 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:57.793293 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:59.793034 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:59.792992 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:10:59.793468 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:59.792992 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:10:59.793468 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:59.793137 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djmtj" podUID="3632857f-077a-4c1e-b064-dcb861c45095" Jul 24 00:10:59.793468 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:10:59.792992 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:10:59.793468 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:59.793208 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rjqbq" podUID="fd6a6183-624e-4420-99cd-df0beaa4ae40" Jul 24 00:10:59.793468 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:10:59.793280 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-thh2r" podUID="833f0c21-b425-49fd-a639-2e2b235f1e7c" Jul 24 00:11:00.552679 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.552601 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-102.ec2.internal" event="NodeReady" Jul 24 00:11:00.552855 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.552765 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 00:11:00.595402 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.594532 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cbc6d6f9c-8w96s"] Jul 24 00:11:00.622013 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.621193 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw"] Jul 24 00:11:00.623233 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.622249 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.625270 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.625246 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 00:11:00.625536 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.625513 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2gxlr\"" Jul 24 00:11:00.625839 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.625816 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 00:11:00.625973 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.625851 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 00:11:00.632759 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.632734 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 00:11:00.633455 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.633273 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8"] Jul 24 00:11:00.634689 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.633981 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.637410 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.637388 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pzd85\"" Jul 24 00:11:00.637583 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.637568 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 00:11:00.637800 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.637769 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.637922 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.637751 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 00:11:00.638157 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.638136 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.647501 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.647466 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-qffj9"] Jul 24 00:11:00.647614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.647600 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.650381 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.650335 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.650533 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.650505 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.650699 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.650682 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5wmbl\"" Jul 24 00:11:00.650765 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.650710 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 00:11:00.651076 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.651054 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 00:11:00.659725 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.659700 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-654cb6b4b5-ps8tt"] Jul 24 00:11:00.659836 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.659823 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.663992 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.663968 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 00:11:00.664105 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.664029 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.664166 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.663970 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.664166 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.664145 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lv9qr\"" Jul 24 00:11:00.664268 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.664237 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 00:11:00.670879 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.670833 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 00:11:00.674114 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.674093 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t"] Jul 24 00:11:00.674230 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.674215 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.677084 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677062 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.677194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677072 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 00:11:00.677194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677111 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 00:11:00.677442 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677423 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lfqbp\"" Jul 24 00:11:00.677588 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677562 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.677664 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677568 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 00:11:00.677664 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.677600 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 00:11:00.688473 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.688440 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54"] Jul 24 00:11:00.688600 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.688567 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:00.691101 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.691079 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 00:11:00.691219 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.691142 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j6lz4\"" Jul 24 00:11:00.691312 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.691285 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.691525 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.691504 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.699218 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.699191 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8"] Jul 24 00:11:00.699447 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.699426 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.702040 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.702020 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 00:11:00.702040 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.702096 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.702040 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.702156 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 00:11:00.702040 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.702240 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kltg4\"" Jul 24 00:11:00.702040 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.702322 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.715378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.715355 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" Jul 24 00:11:00.715507 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.715357 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-chwt8"] Jul 24 00:11:00.717958 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.717938 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.718068 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.718040 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2h4xx\"" Jul 24 00:11:00.718123 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.718044 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.725636 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.725611 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr"] Jul 24 00:11:00.725774 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.725753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.728107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.728086 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 00:11:00.728835 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.728812 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.728953 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.728812 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 00:11:00.728953 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.728913 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.729070 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.728997 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l46t4\"" Jul 24 00:11:00.733760 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.733740 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 00:11:00.739983 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.739960 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jh5zk"] Jul 24 00:11:00.740089 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.740074 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" Jul 24 00:11:00.744014 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.743991 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.744179 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.744084 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.744179 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.744164 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-n4n6w\"" Jul 24 00:11:00.754549 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754520 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw"] Jul 24 00:11:00.754549 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754555 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8"] Jul 24 00:11:00.754727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754578 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t"] Jul 24 00:11:00.754727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754602 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-qffj9"] Jul 24 00:11:00.754727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754621 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cbc6d6f9c-8w96s"] Jul 24 00:11:00.754727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754639 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8"] Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754742 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754762 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-chwt8"] Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754794 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54"] Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754816 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jh5zk"] Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754834 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-654cb6b4b5-ps8tt"] Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754876 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr"] Jul 24 00:11:00.754914 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.754914 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nmfqx"] Jul 24 00:11:00.757600 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.757576 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 00:11:00.758037 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.757652 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 00:11:00.758037 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.757885 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qlhxg\"" Jul 24 00:11:00.758166 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.758148 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 00:11:00.768535 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768511 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-service-ca-bundle\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.768663 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-serving-cert\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.768663 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/230ddb24-8a95-4058-b34a-25b6d01076d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.768663 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768605 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-image-registry-private-configuration\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.768663 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768633 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.768663 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768659 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768673 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-stats-auth\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768694 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768775 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhqv\" (UniqueName: \"kubernetes.io/projected/5731834c-6e4d-474c-91d3-e92d41b96d00-kube-api-access-klhqv\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768819 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a19ea5b-7f1e-4b45-ba88-e359708a673c-ca-trust-extracted\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768839 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsh4d\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-kube-api-access-xsh4d\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768869 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/230ddb24-8a95-4058-b34a-25b6d01076d1-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768894 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktlw\" (UniqueName: \"kubernetes.io/projected/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-kube-api-access-lktlw\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768929 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-snapshots\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.768971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768957 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-trusted-ca\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.768978 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-bound-sa-token\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbrxd\" (UniqueName: \"kubernetes.io/projected/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-kube-api-access-mbrxd\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-default-certificate\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769134 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769154 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-installation-pull-secrets\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhk44\" (UniqueName: \"kubernetes.io/projected/230ddb24-8a95-4058-b34a-25b6d01076d1-kube-api-access-mhk44\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769188 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-tmp\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769226 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-certificates\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.769375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.769371 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.772255 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.772230 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nmfqx"] Jul 24 00:11:00.772434 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.772416 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.775088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.775049 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dcszn\"" Jul 24 00:11:00.775420 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.775399 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 00:11:00.775515 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.775492 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 00:11:00.839715 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.839681 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:00.870432 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870395 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-installation-pull-secrets\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.870432 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-tmp\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870462 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.870572 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870595 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4fe5bb-86b1-4155-a8cd-f3a151519316-config-volume\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.870659 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls podName:6d8e7797-9ab7-4b76-a5cf-f6b16f548776 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:01.370635976 +0000 UTC m=+38.103270996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bdrx8" (UID: "6d8e7797-9ab7-4b76-a5cf-f6b16f548776") : secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:00.870687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870678 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-service-ca-bundle\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870714 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-serving-cert\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870743 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/230ddb24-8a95-4058-b34a-25b6d01076d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870796 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4fe5bb-86b1-4155-a8cd-f3a151519316-tmp-dir\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhqv\" (UniqueName: \"kubernetes.io/projected/5731834c-6e4d-474c-91d3-e92d41b96d00-kube-api-access-klhqv\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-stats-auth\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870929 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd52790b-d34a-4c5b-8bf9-b1170f865154-config\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870952 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-config\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871014 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g82d6\" (UniqueName: \"kubernetes.io/projected/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-kube-api-access-g82d6\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871039 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktlw\" (UniqueName: \"kubernetes.io/projected/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-kube-api-access-lktlw\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.871088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28t7\" (UniqueName: \"kubernetes.io/projected/0f4fe5bb-86b1-4155-a8cd-f3a151519316-kube-api-access-f28t7\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871133 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871159 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee-cert\") pod \"ingress-canary-jh5zk\" (UID: \"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee\") " pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871195 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhk44\" (UniqueName: \"kubernetes.io/projected/230ddb24-8a95-4058-b34a-25b6d01076d1-kube-api-access-mhk44\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.870576 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.870930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-tmp\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-certificates\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.871227 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbc6d6f9c-8w96s: secret "image-registry-tls" not found Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.871328 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls podName:6a19ea5b-7f1e-4b45-ba88-e359708a673c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:01.371302058 +0000 UTC m=+38.103937089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls") pod "image-registry-6cbc6d6f9c-8w96s" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c") : secret "image-registry-tls" not found Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-service-ca-bundle\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd52790b-d34a-4c5b-8bf9-b1170f865154-trusted-ca\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871582 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-image-registry-private-configuration\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871650 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2r8\" (UniqueName: \"kubernetes.io/projected/07fe13fd-0f87-4308-bd13-1b0fab1394bf-kube-api-access-xx2r8\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:00.871694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871679 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a19ea5b-7f1e-4b45-ba88-e359708a673c-ca-trust-extracted\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsh4d\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-kube-api-access-xsh4d\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871734 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5m8\" (UniqueName: \"kubernetes.io/projected/8e998951-5d35-4949-b85f-1ebe6b302c3d-kube-api-access-xq5m8\") pod \"volume-data-source-validator-549c6d54cf-8hxz8\" (UID: \"8e998951-5d35-4949-b85f-1ebe6b302c3d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/230ddb24-8a95-4058-b34a-25b6d01076d1-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-snapshots\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-trusted-ca\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871807 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd52790b-d34a-4c5b-8bf9-b1170f865154-serving-cert\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871805 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-certificates\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.871832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-bound-sa-token\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.871991 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.872066 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:01.372047278 +0000 UTC m=+38.104682295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : secret "router-metrics-certs-default" not found Jul 24 00:11:00.872327 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.872171 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:01.37215959 +0000 UTC m=+38.104794624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : configmap references non-existent config key: service-ca.crt Jul 24 00:11:00.872762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.872444 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a19ea5b-7f1e-4b45-ba88-e359708a673c-ca-trust-extracted\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.872762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.872536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.873257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-snapshots\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.873257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/230ddb24-8a95-4058-b34a-25b6d01076d1-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.873257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4vw2\" (UniqueName: \"kubernetes.io/projected/fd52790b-d34a-4c5b-8bf9-b1170f865154-kube-api-access-p4vw2\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.873257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873255 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlqlw\" (UniqueName: \"kubernetes.io/projected/7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee-kube-api-access-dlqlw\") pod \"ingress-canary-jh5zk\" (UID: \"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee\") " pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.873551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873299 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbrxd\" (UniqueName: \"kubernetes.io/projected/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-kube-api-access-mbrxd\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.873551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873319 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-trusted-ca\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.873551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-default-certificate\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.873551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.873551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.873411 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk6xk\" (UniqueName: \"kubernetes.io/projected/be0c4444-3a45-4921-8e3a-c7733c53b242-kube-api-access-jk6xk\") pod \"network-check-source-7d88cb55d8-7cgsr\" (UID: \"be0c4444-3a45-4921-8e3a-c7733c53b242\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" Jul 24 00:11:00.876093 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.876063 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/230ddb24-8a95-4058-b34a-25b6d01076d1-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.876237 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.876100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-installation-pull-secrets\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.876237 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.876120 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-serving-cert\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.876237 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.876176 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-image-registry-private-configuration\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.882167 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.882133 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-stats-auth\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.882290 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.882223 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-default-certificate\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.892378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.892317 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhqv\" (UniqueName: \"kubernetes.io/projected/5731834c-6e4d-474c-91d3-e92d41b96d00-kube-api-access-klhqv\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:00.892546 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.892522 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktlw\" (UniqueName: \"kubernetes.io/projected/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-kube-api-access-lktlw\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:00.892769 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.892726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbrxd\" (UniqueName: \"kubernetes.io/projected/0bb576d5-b2da-4ff1-9ca6-68f74da0ab02-kube-api-access-mbrxd\") pod \"insights-operator-5885fdb759-qffj9\" (UID: \"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02\") " pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.893321 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.893301 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-bound-sa-token\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.893847 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.893827 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsh4d\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-kube-api-access-xsh4d\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:00.894229 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.894202 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhk44\" (UniqueName: \"kubernetes.io/projected/230ddb24-8a95-4058-b34a-25b6d01076d1-kube-api-access-mhk44\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw\" (UID: \"230ddb24-8a95-4058-b34a-25b6d01076d1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.947194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.947161 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" Jul 24 00:11:00.959697 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.959657 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:00.973943 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.973912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk6xk\" (UniqueName: \"kubernetes.io/projected/be0c4444-3a45-4921-8e3a-c7733c53b242-kube-api-access-jk6xk\") pod \"network-check-source-7d88cb55d8-7cgsr\" (UID: \"be0c4444-3a45-4921-8e3a-c7733c53b242\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" Jul 24 00:11:00.974098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.973988 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4fe5bb-86b1-4155-a8cd-f3a151519316-config-volume\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.974098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974017 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:00.974098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974046 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4fe5bb-86b1-4155-a8cd-f3a151519316-tmp-dir\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.974098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974078 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-qffj9" Jul 24 00:11:00.974098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974087 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd52790b-d34a-4c5b-8bf9-b1170f865154-config\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974145 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-config\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g82d6\" (UniqueName: \"kubernetes.io/projected/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-kube-api-access-g82d6\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974220 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28t7\" (UniqueName: \"kubernetes.io/projected/0f4fe5bb-86b1-4155-a8cd-f3a151519316-kube-api-access-f28t7\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974256 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee-cert\") pod \"ingress-canary-jh5zk\" (UID: \"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee\") " pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974279 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd52790b-d34a-4c5b-8bf9-b1170f865154-trusted-ca\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.974328 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974316 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2r8\" (UniqueName: \"kubernetes.io/projected/07fe13fd-0f87-4308-bd13-1b0fab1394bf-kube-api-access-xx2r8\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:00.974652 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974631 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4fe5bb-86b1-4155-a8cd-f3a151519316-config-volume\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.974922 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974900 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f4fe5bb-86b1-4155-a8cd-f3a151519316-tmp-dir\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.975051 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-config\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.975051 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.974995 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 00:11:00.975051 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.975050 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls podName:07fe13fd-0f87-4308-bd13-1b0fab1394bf nodeName:}" failed. No retries permitted until 2026-07-24 00:11:01.475032266 +0000 UTC m=+38.207667273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls") pod "cluster-samples-operator-6748489656-tgp7t" (UID: "07fe13fd-0f87-4308-bd13-1b0fab1394bf") : secret "samples-operator-tls" not found Jul 24 00:11:00.975219 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.974994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5m8\" (UniqueName: \"kubernetes.io/projected/8e998951-5d35-4949-b85f-1ebe6b302c3d-kube-api-access-xq5m8\") pod \"volume-data-source-validator-549c6d54cf-8hxz8\" (UID: \"8e998951-5d35-4949-b85f-1ebe6b302c3d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" Jul 24 00:11:00.975219 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.975183 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd52790b-d34a-4c5b-8bf9-b1170f865154-serving-cert\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.975219 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.975209 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:11:00.975339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.975227 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4vw2\" (UniqueName: \"kubernetes.io/projected/fd52790b-d34a-4c5b-8bf9-b1170f865154-kube-api-access-p4vw2\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.975339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.975253 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlqlw\" (UniqueName: \"kubernetes.io/projected/7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee-kube-api-access-dlqlw\") pod \"ingress-canary-jh5zk\" (UID: \"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee\") " pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.975339 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:00.975270 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls podName:0f4fe5bb-86b1-4155-a8cd-f3a151519316 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:01.475252945 +0000 UTC m=+38.207887954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls") pod "dns-default-nmfqx" (UID: "0f4fe5bb-86b1-4155-a8cd-f3a151519316") : secret "dns-default-metrics-tls" not found Jul 24 00:11:00.975339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.975319 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.975823 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.975394 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fd52790b-d34a-4c5b-8bf9-b1170f865154-config\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.976046 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.976002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd52790b-d34a-4c5b-8bf9-b1170f865154-trusted-ca\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.978100 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.978076 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:00.978194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.978125 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fd52790b-d34a-4c5b-8bf9-b1170f865154-serving-cert\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.978194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.978170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee-cert\") pod \"ingress-canary-jh5zk\" (UID: \"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee\") " pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.989203 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.989171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4vw2\" (UniqueName: \"kubernetes.io/projected/fd52790b-d34a-4c5b-8bf9-b1170f865154-kube-api-access-p4vw2\") pod \"console-operator-584dcb6b54-chwt8\" (UID: \"fd52790b-d34a-4c5b-8bf9-b1170f865154\") " pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:00.989399 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.989374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5m8\" (UniqueName: \"kubernetes.io/projected/8e998951-5d35-4949-b85f-1ebe6b302c3d-kube-api-access-xq5m8\") pod \"volume-data-source-validator-549c6d54cf-8hxz8\" (UID: \"8e998951-5d35-4949-b85f-1ebe6b302c3d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" Jul 24 00:11:00.989531 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.989505 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28t7\" (UniqueName: \"kubernetes.io/projected/0f4fe5bb-86b1-4155-a8cd-f3a151519316-kube-api-access-f28t7\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:00.989623 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.989607 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk6xk\" (UniqueName: \"kubernetes.io/projected/be0c4444-3a45-4921-8e3a-c7733c53b242-kube-api-access-jk6xk\") pod \"network-check-source-7d88cb55d8-7cgsr\" (UID: \"be0c4444-3a45-4921-8e3a-c7733c53b242\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" Jul 24 00:11:00.989687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.989647 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlqlw\" (UniqueName: \"kubernetes.io/projected/7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee-kube-api-access-dlqlw\") pod \"ingress-canary-jh5zk\" (UID: \"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee\") " pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:00.989735 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.989685 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2r8\" (UniqueName: \"kubernetes.io/projected/07fe13fd-0f87-4308-bd13-1b0fab1394bf-kube-api-access-xx2r8\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:00.990821 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:00.990793 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g82d6\" (UniqueName: \"kubernetes.io/projected/af80fcdb-8d4b-4585-93e5-032d9b1b8ac1-kube-api-access-g82d6\") pod \"service-ca-operator-5d6c75f84f-rzb54\" (UID: \"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:01.008854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.008821 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" Jul 24 00:11:01.025513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.025487 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" Jul 24 00:11:01.036290 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.036258 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:01.049320 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.049284 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" Jul 24 00:11:01.066498 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.066454 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jh5zk" Jul 24 00:11:01.105804 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.105717 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:01.169537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.169499 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:01.378802 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.378719 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:01.378802 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.378799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.378832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.378929 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:02.378908329 +0000 UTC m=+39.111543351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : configmap references non-existent config key: service-ca.crt Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.378954 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.378969 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbc6d6f9c-8w96s: secret "image-registry-tls" not found Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.378996 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.379003 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls podName:6a19ea5b-7f1e-4b45-ba88-e359708a673c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:02.378990811 +0000 UTC m=+39.111625828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls") pod "image-registry-6cbc6d6f9c-8w96s" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c") : secret "image-registry-tls" not found Jul 24 00:11:01.379049 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.378969 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:01.379337 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.379086 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:11:01.379337 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.379109 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls podName:6d8e7797-9ab7-4b76-a5cf-f6b16f548776 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:02.379089491 +0000 UTC m=+39.111724513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bdrx8" (UID: "6d8e7797-9ab7-4b76-a5cf-f6b16f548776") : secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:01.379337 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.379134 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:02.379125159 +0000 UTC m=+39.111760165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : secret "router-metrics-certs-default" not found Jul 24 00:11:01.480334 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.480290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:01.480519 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.480432 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 00:11:01.480519 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.480437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:01.480519 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.480502 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls podName:07fe13fd-0f87-4308-bd13-1b0fab1394bf nodeName:}" failed. No retries permitted until 2026-07-24 00:11:02.480484674 +0000 UTC m=+39.213119677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls") pod "cluster-samples-operator-6748489656-tgp7t" (UID: "07fe13fd-0f87-4308-bd13-1b0fab1394bf") : secret "samples-operator-tls" not found Jul 24 00:11:01.480519 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.480516 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:11:01.480729 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:01.480571 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls podName:0f4fe5bb-86b1-4155-a8cd-f3a151519316 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:02.480555456 +0000 UTC m=+39.213190488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls") pod "dns-default-nmfqx" (UID: "0f4fe5bb-86b1-4155-a8cd-f3a151519316") : secret "dns-default-metrics-tls" not found Jul 24 00:11:01.794580 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.793355 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:11:01.794580 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.794541 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:11:01.794976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.794933 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:11:01.799964 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.799001 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-79cjj\"" Jul 24 00:11:01.799964 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.799252 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 00:11:01.799964 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.799532 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 00:11:01.799964 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.799594 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvqg9\"" Jul 24 00:11:01.839431 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.839403 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr"] Jul 24 00:11:01.845744 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.845713 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jh5zk"] Jul 24 00:11:01.848903 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:01.848880 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0f4562_f05a_4a3c_9e4b_c45eda7ffaee.slice/crio-2c0f4922a4b10dd6eadfe1f0b97428e5644e87c69480197d384461ca396dbb26 WatchSource:0}: Error finding container 2c0f4922a4b10dd6eadfe1f0b97428e5644e87c69480197d384461ca396dbb26: Status 404 returned error can't find the container with id 2c0f4922a4b10dd6eadfe1f0b97428e5644e87c69480197d384461ca396dbb26 Jul 24 00:11:01.854553 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.854527 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54"] Jul 24 00:11:01.857178 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.857127 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8"] Jul 24 00:11:01.857235 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:01.857201 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf80fcdb_8d4b_4585_93e5_032d9b1b8ac1.slice/crio-9c4e82b89cc43943651571cdf6fd4abcc35feaf0c8966fb2e236d5bfa83d7ef7 WatchSource:0}: Error finding container 9c4e82b89cc43943651571cdf6fd4abcc35feaf0c8966fb2e236d5bfa83d7ef7: Status 404 returned error can't find the container with id 9c4e82b89cc43943651571cdf6fd4abcc35feaf0c8966fb2e236d5bfa83d7ef7 Jul 24 00:11:01.859426 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:01.859402 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e998951_5d35_4949_b85f_1ebe6b302c3d.slice/crio-a5579e5f1b11c59ee4186a27f5cf157e6ffc21b5a6eb1d99138c0bd9f436ff8d WatchSource:0}: Error finding container a5579e5f1b11c59ee4186a27f5cf157e6ffc21b5a6eb1d99138c0bd9f436ff8d: Status 404 returned error can't find the container with id a5579e5f1b11c59ee4186a27f5cf157e6ffc21b5a6eb1d99138c0bd9f436ff8d Jul 24 00:11:01.862518 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.862492 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-chwt8"] Jul 24 00:11:01.869973 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.865314 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw"] Jul 24 00:11:01.869973 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:01.866562 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd52790b_d34a_4c5b_8bf9_b1170f865154.slice/crio-c899a37aac23d7e62ac8b84115f6dea1479fdc2f05a32ad9ac8791e4388b3838 WatchSource:0}: Error finding container c899a37aac23d7e62ac8b84115f6dea1479fdc2f05a32ad9ac8791e4388b3838: Status 404 returned error can't find the container with id c899a37aac23d7e62ac8b84115f6dea1479fdc2f05a32ad9ac8791e4388b3838 Jul 24 00:11:01.871319 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:01.871294 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230ddb24_8a95_4058_b34a_25b6d01076d1.slice/crio-c62264766526e418de6789a9ebf1a7f04e6799ec77e38e69d4eacfead0a9e481 WatchSource:0}: Error finding container c62264766526e418de6789a9ebf1a7f04e6799ec77e38e69d4eacfead0a9e481: Status 404 returned error can't find the container with id c62264766526e418de6789a9ebf1a7f04e6799ec77e38e69d4eacfead0a9e481 Jul 24 00:11:01.871809 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:01.871779 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-qffj9"] Jul 24 00:11:01.874475 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:01.874452 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb576d5_b2da_4ff1_9ca6_68f74da0ab02.slice/crio-730a3810a567d7c3ac4f8f5dba4805a6b2b0af7b97ca620812c4ca13936647a5 WatchSource:0}: Error finding container 730a3810a567d7c3ac4f8f5dba4805a6b2b0af7b97ca620812c4ca13936647a5: Status 404 returned error can't find the container with id 730a3810a567d7c3ac4f8f5dba4805a6b2b0af7b97ca620812c4ca13936647a5 Jul 24 00:11:02.026681 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.026511 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:02.042969 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.042938 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" event={"ID":"fd52790b-d34a-4c5b-8bf9-b1170f865154","Type":"ContainerStarted","Data":"c899a37aac23d7e62ac8b84115f6dea1479fdc2f05a32ad9ac8791e4388b3838"} Jul 24 00:11:02.044159 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.044126 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" event={"ID":"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1","Type":"ContainerStarted","Data":"9c4e82b89cc43943651571cdf6fd4abcc35feaf0c8966fb2e236d5bfa83d7ef7"} Jul 24 00:11:02.045222 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.045192 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" event={"ID":"230ddb24-8a95-4058-b34a-25b6d01076d1","Type":"ContainerStarted","Data":"c62264766526e418de6789a9ebf1a7f04e6799ec77e38e69d4eacfead0a9e481"} Jul 24 00:11:02.048763 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.048733 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jh5zk" event={"ID":"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee","Type":"ContainerStarted","Data":"2c0f4922a4b10dd6eadfe1f0b97428e5644e87c69480197d384461ca396dbb26"} Jul 24 00:11:02.050229 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.050199 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qffj9" event={"ID":"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02","Type":"ContainerStarted","Data":"730a3810a567d7c3ac4f8f5dba4805a6b2b0af7b97ca620812c4ca13936647a5"} Jul 24 00:11:02.051416 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.051395 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" event={"ID":"8e998951-5d35-4949-b85f-1ebe6b302c3d","Type":"ContainerStarted","Data":"a5579e5f1b11c59ee4186a27f5cf157e6ffc21b5a6eb1d99138c0bd9f436ff8d"} Jul 24 00:11:02.054130 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.054098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerStarted","Data":"e23d28b2b4c4b54383ded96f5ffb6976bc1fdffa16264d1aa68593f87eccce91"} Jul 24 00:11:02.055229 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.055205 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" event={"ID":"be0c4444-3a45-4921-8e3a-c7733c53b242","Type":"ContainerStarted","Data":"3408303fc7c23b62d20a9c262b725113de827d59f7009ddb9a3082639c66984b"} Jul 24 00:11:02.098833 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.098798 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:02.389644 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.389598 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:02.389822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.389678 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:02.389822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.389708 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:02.389822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.389750 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:02.389822 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389784 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:04.389762224 +0000 UTC m=+41.122397251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : configmap references non-existent config key: service-ca.crt Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389847 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389880 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389900 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls podName:6d8e7797-9ab7-4b76-a5cf-f6b16f548776 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:04.389890072 +0000 UTC m=+41.122525076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bdrx8" (UID: "6d8e7797-9ab7-4b76-a5cf-f6b16f548776") : secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389904 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbc6d6f9c-8w96s: secret "image-registry-tls" not found Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389947 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389959 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls podName:6a19ea5b-7f1e-4b45-ba88-e359708a673c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:04.389940813 +0000 UTC m=+41.122575847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls") pod "image-registry-6cbc6d6f9c-8w96s" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c") : secret "image-registry-tls" not found Jul 24 00:11:02.390040 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.389978 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:04.389965929 +0000 UTC m=+41.122600933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : secret "router-metrics-certs-default" not found Jul 24 00:11:02.491109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.491071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:02.491254 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:02.491131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:02.491254 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.491224 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 00:11:02.491345 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.491291 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:11:02.491345 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.491302 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls podName:07fe13fd-0f87-4308-bd13-1b0fab1394bf nodeName:}" failed. No retries permitted until 2026-07-24 00:11:04.491280956 +0000 UTC m=+41.223915973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls") pod "cluster-samples-operator-6748489656-tgp7t" (UID: "07fe13fd-0f87-4308-bd13-1b0fab1394bf") : secret "samples-operator-tls" not found Jul 24 00:11:02.491450 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:02.491363 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls podName:0f4fe5bb-86b1-4155-a8cd-f3a151519316 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:04.491344435 +0000 UTC m=+41.223979439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls") pod "dns-default-nmfqx" (UID: "0f4fe5bb-86b1-4155-a8cd-f3a151519316") : secret "dns-default-metrics-tls" not found Jul 24 00:11:03.069551 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:03.069509 2585 generic.go:356] "Generic (PLEG): container finished" podID="acb33309-ccf8-4df1-a74a-2f53fc79bee3" containerID="e23d28b2b4c4b54383ded96f5ffb6976bc1fdffa16264d1aa68593f87eccce91" exitCode=0 Jul 24 00:11:03.070115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:03.069583 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerDied","Data":"e23d28b2b4c4b54383ded96f5ffb6976bc1fdffa16264d1aa68593f87eccce91"} Jul 24 00:11:04.010526 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.010453 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:04.080266 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.080229 2585 generic.go:356] "Generic (PLEG): container finished" podID="acb33309-ccf8-4df1-a74a-2f53fc79bee3" containerID="313935cc756efe2d32df625dcc2e702796385e6cbf99373c25c7b592b4d67ba3" exitCode=0 Jul 24 00:11:04.080757 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.080281 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerDied","Data":"313935cc756efe2d32df625dcc2e702796385e6cbf99373c25c7b592b4d67ba3"} Jul 24 00:11:04.410944 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.410898 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:04.411172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.410962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:04.411172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.411018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:04.411172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.411117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411334 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411354 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbc6d6f9c-8w96s: secret "image-registry-tls" not found Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411353 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411415 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls podName:6a19ea5b-7f1e-4b45-ba88-e359708a673c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:08.411394397 +0000 UTC m=+45.144029401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls") pod "image-registry-6cbc6d6f9c-8w96s" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c") : secret "image-registry-tls" not found Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411433 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:08.411423639 +0000 UTC m=+45.144058657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : secret "router-metrics-certs-default" not found Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411458 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:04.411798 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.411500 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls podName:6d8e7797-9ab7-4b76-a5cf-f6b16f548776 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:08.411481819 +0000 UTC m=+45.144116836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bdrx8" (UID: "6d8e7797-9ab7-4b76-a5cf-f6b16f548776") : secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:04.412389 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.412363 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:08.412346518 +0000 UTC m=+45.144981524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : configmap references non-existent config key: service-ca.crt Jul 24 00:11:04.511822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.511782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:04.512001 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.511846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:04.512001 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.511975 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 00:11:04.512124 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.512040 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:11:04.512124 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.512048 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls podName:07fe13fd-0f87-4308-bd13-1b0fab1394bf nodeName:}" failed. No retries permitted until 2026-07-24 00:11:08.512028094 +0000 UTC m=+45.244663110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls") pod "cluster-samples-operator-6748489656-tgp7t" (UID: "07fe13fd-0f87-4308-bd13-1b0fab1394bf") : secret "samples-operator-tls" not found Jul 24 00:11:04.512124 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:04.512091 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls podName:0f4fe5bb-86b1-4155-a8cd-f3a151519316 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:08.512075621 +0000 UTC m=+45.244710630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls") pod "dns-default-nmfqx" (UID: "0f4fe5bb-86b1-4155-a8cd-f3a151519316") : secret "dns-default-metrics-tls" not found Jul 24 00:11:04.980680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:04.980598 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:05.080836 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:05.080788 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:05.084541 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:05.084503 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" event={"ID":"8e998951-5d35-4949-b85f-1ebe6b302c3d","Type":"ContainerStarted","Data":"3925a7e0d7097d1b85af561c4dc9c12328b0f638eec9c8dab79a22a86b6acf27"} Jul 24 00:11:05.102268 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:05.102189 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8hxz8" podStartSLOduration=8.032478264 podStartE2EDuration="10.102167506s" totalImagesPullingTime="2.069689242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:05.10097036 +0000 UTC m=+41.833605377" watchObservedRunningTime="2026-07-24 00:11:05.102167506 +0000 UTC m=+41.834802531" Jul 24 00:11:05.189800 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:05.189709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:05.263928 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:05.263887 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:08.447877 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:08.447820 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448020 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.447993468 +0000 UTC m=+53.180628474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : configmap references non-existent config key: service-ca.crt Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:08.448088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:08.448124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448232 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448244 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbc6d6f9c-8w96s: secret "image-registry-tls" not found Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448250 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:08.448282 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448311 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls podName:6d8e7797-9ab7-4b76-a5cf-f6b16f548776 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.448293708 +0000 UTC m=+53.180928731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bdrx8" (UID: "6d8e7797-9ab7-4b76-a5cf-f6b16f548776") : secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448325 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448340 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls podName:6a19ea5b-7f1e-4b45-ba88-e359708a673c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.448329256 +0000 UTC m=+53.180964271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls") pod "image-registry-6cbc6d6f9c-8w96s" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c") : secret "image-registry-tls" not found Jul 24 00:11:08.448404 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.448359 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.448346651 +0000 UTC m=+53.180981657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : secret "router-metrics-certs-default" not found Jul 24 00:11:08.549491 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:08.549451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:08.549676 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:08.549508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:08.549676 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.549611 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:11:08.549676 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.549667 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls podName:0f4fe5bb-86b1-4155-a8cd-f3a151519316 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.549653089 +0000 UTC m=+53.282288092 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls") pod "dns-default-nmfqx" (UID: "0f4fe5bb-86b1-4155-a8cd-f3a151519316") : secret "dns-default-metrics-tls" not found Jul 24 00:11:08.549815 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.549611 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 00:11:08.549815 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:08.549754 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls podName:07fe13fd-0f87-4308-bd13-1b0fab1394bf nodeName:}" failed. No retries permitted until 2026-07-24 00:11:16.54974029 +0000 UTC m=+53.282375307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls") pod "cluster-samples-operator-6748489656-tgp7t" (UID: "07fe13fd-0f87-4308-bd13-1b0fab1394bf") : secret "samples-operator-tls" not found Jul 24 00:11:09.420145 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.420104 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c"] Jul 24 00:11:09.441593 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.441561 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c"] Jul 24 00:11:09.441748 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.441677 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:09.444435 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.444409 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 00:11:09.444846 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.444830 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 00:11:09.445001 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.444984 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sl59f\"" Jul 24 00:11:09.457160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.457087 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:09.457414 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.457152 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca23905c-87a5-4884-ba50-74e081b46cea-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:09.557722 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.557692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:09.557856 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.557756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca23905c-87a5-4884-ba50-74e081b46cea-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:09.557856 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:09.557840 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:11:09.557856 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:09.557938 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert podName:ca23905c-87a5-4884-ba50-74e081b46cea nodeName:}" failed. No retries permitted until 2026-07-24 00:11:10.057918385 +0000 UTC m=+46.790553388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-tgg5c" (UID: "ca23905c-87a5-4884-ba50-74e081b46cea") : secret "networking-console-plugin-cert" not found Jul 24 00:11:09.558651 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.558624 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ca23905c-87a5-4884-ba50-74e081b46cea-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:09.751113 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.751030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:09.841032 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:09.840993 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.062782 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.062749 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:10.062997 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:10.062921 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:11:10.062997 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:10.062981 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert podName:ca23905c-87a5-4884-ba50-74e081b46cea nodeName:}" failed. No retries permitted until 2026-07-24 00:11:11.062965925 +0000 UTC m=+47.795600932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-tgg5c" (UID: "ca23905c-87a5-4884-ba50-74e081b46cea") : secret "networking-console-plugin-cert" not found Jul 24 00:11:10.099724 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.099689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" event={"ID":"230ddb24-8a95-4058-b34a-25b6d01076d1","Type":"ContainerStarted","Data":"1d306cbf136f2d854b58231be5de4615ba5a1165e904dbba39d37729cdcb5a4d"} Jul 24 00:11:10.101812 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.101787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jh5zk" event={"ID":"7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee","Type":"ContainerStarted","Data":"93888e6dde1c1b5701c2f1c53c8c9e24309cb3cd5d94dbcd541e6a13c9a1a2d5"} Jul 24 00:11:10.103566 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.103539 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qffj9" event={"ID":"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02","Type":"ContainerStarted","Data":"e9b302612e80c488915c1c5abfa5d06eabd45f36adb18214e7a865e5a42a606e"} Jul 24 00:11:10.107352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.107326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdltg" event={"ID":"acb33309-ccf8-4df1-a74a-2f53fc79bee3","Type":"ContainerStarted","Data":"09d59416ca55468be8a3c18562a8e0cb328cd6587a6267c8a5d61ccc7314d93f"} Jul 24 00:11:10.109411 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.109376 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" event={"ID":"be0c4444-3a45-4921-8e3a-c7733c53b242","Type":"ContainerStarted","Data":"9b656f8ffc13e0220745dfc0687e5c9f4f5c836564ff9500970005389f630017"} Jul 24 00:11:10.111107 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.111085 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/0.log" Jul 24 00:11:10.111205 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.111122 2585 generic.go:356] "Generic (PLEG): container finished" podID="fd52790b-d34a-4c5b-8bf9-b1170f865154" containerID="36c616694cd5523fa8faacf10d39c8d753739ebb1c379d2ebd520e181bde72d7" exitCode=255 Jul 24 00:11:10.111347 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.111327 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" event={"ID":"fd52790b-d34a-4c5b-8bf9-b1170f865154","Type":"ContainerDied","Data":"36c616694cd5523fa8faacf10d39c8d753739ebb1c379d2ebd520e181bde72d7"} Jul 24 00:11:10.111470 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.111409 2585 scope.go:117] "RemoveContainer" containerID="36c616694cd5523fa8faacf10d39c8d753739ebb1c379d2ebd520e181bde72d7" Jul 24 00:11:10.115131 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.115102 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" event={"ID":"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1","Type":"ContainerStarted","Data":"9801bea1ae7aedead4a9d43f1eb50d904e63a6d6e2259cdc762d0ba8d5b8af45"} Jul 24 00:11:10.145923 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.145608 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tdltg" podStartSLOduration=13.031896116 podStartE2EDuration="46.145582875s" totalImagesPullingTime="33.113686759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:10.143552573 +0000 UTC m=+46.876187598" watchObservedRunningTime="2026-07-24 00:11:10.145582875 +0000 UTC m=+46.878217901" Jul 24 00:11:10.145923 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.145785 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" podStartSLOduration=7.459006235 podStartE2EDuration="15.145775892s" totalImagesPullingTime="7.686769657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:10.118248536 +0000 UTC m=+46.850883593" watchObservedRunningTime="2026-07-24 00:11:10.145775892 +0000 UTC m=+46.878410897" Jul 24 00:11:10.192230 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.192179 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jh5zk" podStartSLOduration=2.475687843 podStartE2EDuration="10.192158907s" totalImagesPullingTime="7.716471064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:10.190647416 +0000 UTC m=+46.923282455" watchObservedRunningTime="2026-07-24 00:11:10.192158907 +0000 UTC m=+46.924793932" Jul 24 00:11:10.217510 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.217436 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-qffj9" podStartSLOduration=7.534407329 podStartE2EDuration="15.21741704s" totalImagesPullingTime="7.683009711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:10.217405994 +0000 UTC m=+46.950041020" watchObservedRunningTime="2026-07-24 00:11:10.21741704 +0000 UTC m=+46.950052067" Jul 24 00:11:10.243427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.242388 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" podStartSLOduration=7.54119668 podStartE2EDuration="15.242367313s" totalImagesPullingTime="7.701170633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:10.241587897 +0000 UTC m=+46.974222923" watchObservedRunningTime="2026-07-24 00:11:10.242367313 +0000 UTC m=+46.975002340" Jul 24 00:11:10.265636 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.265368 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-7cgsr" podStartSLOduration=7.5518140339999995 podStartE2EDuration="15.265347517s" totalImagesPullingTime="7.713533483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:10.265153598 +0000 UTC m=+46.997788624" watchObservedRunningTime="2026-07-24 00:11:10.265347517 +0000 UTC m=+46.997982544" Jul 24 00:11:10.271824 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.271786 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.337292 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.337189 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.488802 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.488745 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.588932 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.588824 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.797115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.797080 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.823836 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.823802 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p"] Jul 24 00:11:10.838216 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.838184 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" Jul 24 00:11:10.839237 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.839122 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p"] Jul 24 00:11:10.841049 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.841018 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 00:11:10.841049 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.841022 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4dc76\"" Jul 24 00:11:10.841454 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.841433 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 00:11:10.872014 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.871974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ln7\" (UniqueName: \"kubernetes.io/projected/305fa5cc-79a6-4f1f-8d86-39bbf6633f1b-kube-api-access-88ln7\") pod \"migrator-6865c9967-zn62p\" (UID: \"305fa5cc-79a6-4f1f-8d86-39bbf6633f1b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" Jul 24 00:11:10.897351 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.897312 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:10.973300 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.973269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ln7\" (UniqueName: \"kubernetes.io/projected/305fa5cc-79a6-4f1f-8d86-39bbf6633f1b-kube-api-access-88ln7\") pod \"migrator-6865c9967-zn62p\" (UID: \"305fa5cc-79a6-4f1f-8d86-39bbf6633f1b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" Jul 24 00:11:10.984483 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.984446 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ln7\" (UniqueName: \"kubernetes.io/projected/305fa5cc-79a6-4f1f-8d86-39bbf6633f1b-kube-api-access-88ln7\") pod \"migrator-6865c9967-zn62p\" (UID: \"305fa5cc-79a6-4f1f-8d86-39bbf6633f1b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" Jul 24 00:11:10.999371 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:10.999341 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.037002 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.036956 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:11.037158 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.037028 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:11.074568 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.074527 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:11.074730 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.074691 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:11:11.074772 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.074759 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert podName:ca23905c-87a5-4884-ba50-74e081b46cea nodeName:}" failed. No retries permitted until 2026-07-24 00:11:13.074742754 +0000 UTC m=+49.807377758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-tgg5c" (UID: "ca23905c-87a5-4884-ba50-74e081b46cea") : secret "networking-console-plugin-cert" not found Jul 24 00:11:11.120613 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.120537 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/1.log" Jul 24 00:11:11.120976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.120963 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/0.log" Jul 24 00:11:11.121045 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.120994 2585 generic.go:356] "Generic (PLEG): container finished" podID="fd52790b-d34a-4c5b-8bf9-b1170f865154" containerID="3d7c69d881ae7793e257892e5977eb7bbf245aea56952b88cfb469975f49dc77" exitCode=255 Jul 24 00:11:11.121081 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.121073 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" event={"ID":"fd52790b-d34a-4c5b-8bf9-b1170f865154","Type":"ContainerDied","Data":"3d7c69d881ae7793e257892e5977eb7bbf245aea56952b88cfb469975f49dc77"} Jul 24 00:11:11.121124 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.121111 2585 scope.go:117] "RemoveContainer" containerID="36c616694cd5523fa8faacf10d39c8d753739ebb1c379d2ebd520e181bde72d7" Jul 24 00:11:11.121376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.121324 2585 scope.go:117] "RemoveContainer" containerID="3d7c69d881ae7793e257892e5977eb7bbf245aea56952b88cfb469975f49dc77" Jul 24 00:11:11.121549 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.121523 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-chwt8_openshift-console-operator(fd52790b-d34a-4c5b-8bf9-b1170f865154)\"" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" podUID="fd52790b-d34a-4c5b-8bf9-b1170f865154" Jul 24 00:11:11.140599 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.140561 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.150908 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.150880 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" Jul 24 00:11:11.198100 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.198070 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.315757 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.315733 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p"] Jul 24 00:11:11.318003 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:11.317966 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305fa5cc_79a6_4f1f_8d86_39bbf6633f1b.slice/crio-74de3f5ff2378cbc9ff35235096d19e35919ae92a27fc777fa5bfdafd35a22f5 WatchSource:0}: Error finding container 74de3f5ff2378cbc9ff35235096d19e35919ae92a27fc777fa5bfdafd35a22f5: Status 404 returned error can't find the container with id 74de3f5ff2378cbc9ff35235096d19e35919ae92a27fc777fa5bfdafd35a22f5 Jul 24 00:11:11.448852 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.448772 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p9zgc"] Jul 24 00:11:11.479714 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.479679 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p9zgc"] Jul 24 00:11:11.479903 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.479782 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.482901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.482557 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nsh42\"" Jul 24 00:11:11.482901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.482675 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 00:11:11.482901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.482847 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 00:11:11.501808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.501773 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.575667 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.575633 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.580835 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.580810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbgj\" (UniqueName: \"kubernetes.io/projected/24722498-2095-48cc-941a-c82ef22e9ff1-kube-api-access-ljbgj\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.580931 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.580846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24722498-2095-48cc-941a-c82ef22e9ff1-crio-socket\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.580931 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.580897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.580997 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.580930 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.581102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.581082 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24722498-2095-48cc-941a-c82ef22e9ff1-data-volume\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.674181 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.674141 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.682373 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682343 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbgj\" (UniqueName: \"kubernetes.io/projected/24722498-2095-48cc-941a-c82ef22e9ff1-kube-api-access-ljbgj\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.682464 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24722498-2095-48cc-941a-c82ef22e9ff1-crio-socket\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.682464 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.682464 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682445 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.682557 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24722498-2095-48cc-941a-c82ef22e9ff1-data-volume\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.682596 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.682574 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:11:11.682648 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.682635 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:12.182619101 +0000 UTC m=+48.915254109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : secret "insights-runtime-extractor-tls" not found Jul 24 00:11:11.682705 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.682632 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 00:11:11.682705 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24722498-2095-48cc-941a-c82ef22e9ff1-crio-socket\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.682705 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:11.682678 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:12.182668716 +0000 UTC m=+48.915303729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : configmap "kube-rbac-proxy" not found Jul 24 00:11:11.682853 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.682839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24722498-2095-48cc-941a-c82ef22e9ff1-data-volume\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.693145 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.693119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbgj\" (UniqueName: \"kubernetes.io/projected/24722498-2095-48cc-941a-c82ef22e9ff1-kube-api-access-ljbgj\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:11.851407 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.851373 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:11.950358 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:11.950325 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:12.125979 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.125908 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/1.log" Jul 24 00:11:12.126293 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.126263 2585 scope.go:117] "RemoveContainer" containerID="3d7c69d881ae7793e257892e5977eb7bbf245aea56952b88cfb469975f49dc77" Jul 24 00:11:12.126476 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:12.126455 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-chwt8_openshift-console-operator(fd52790b-d34a-4c5b-8bf9-b1170f865154)\"" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" podUID="fd52790b-d34a-4c5b-8bf9-b1170f865154" Jul 24 00:11:12.127119 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.127098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" event={"ID":"305fa5cc-79a6-4f1f-8d86-39bbf6633f1b","Type":"ContainerStarted","Data":"74de3f5ff2378cbc9ff35235096d19e35919ae92a27fc777fa5bfdafd35a22f5"} Jul 24 00:11:12.187566 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.187518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:12.187722 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.187597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:12.187722 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:12.187653 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:11:12.187855 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:12.187726 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:13.187709858 +0000 UTC m=+49.920344861 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : secret "insights-runtime-extractor-tls" not found Jul 24 00:11:12.187855 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:12.187731 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 00:11:12.187855 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:12.187807 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:13.187788094 +0000 UTC m=+49.920423100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : configmap "kube-rbac-proxy" not found Jul 24 00:11:12.878744 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.878709 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:12.984991 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:12.984949 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:13.097030 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.096992 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:13.097176 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:13.097154 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:11:13.097228 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:13.097223 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert podName:ca23905c-87a5-4884-ba50-74e081b46cea nodeName:}" failed. No retries permitted until 2026-07-24 00:11:17.097205274 +0000 UTC m=+53.829840278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-tgg5c" (UID: "ca23905c-87a5-4884-ba50-74e081b46cea") : secret "networking-console-plugin-cert" not found Jul 24 00:11:13.150617 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.150537 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:13.197845 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.197810 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:13.198011 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.197855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:13.198011 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:13.197981 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:11:13.198011 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:13.197998 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 00:11:13.198110 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:13.198063 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:15.198039963 +0000 UTC m=+51.930674985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : configmap "kube-rbac-proxy" not found Jul 24 00:11:13.198110 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:13.198086 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:15.198074405 +0000 UTC m=+51.930709420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : secret "insights-runtime-extractor-tls" not found Jul 24 00:11:13.299434 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.299402 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:13.455439 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.455421 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjx9t_ee0c4f9c-792d-4b61-9493-44a83f3632ba/dns-node-resolver/0.log" Jul 24 00:11:13.460147 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.460124 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:13.558770 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.558731 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:13.937808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:13.937771 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:14.036372 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.036336 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:14.137910 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.137855 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" event={"ID":"305fa5cc-79a6-4f1f-8d86-39bbf6633f1b","Type":"ContainerStarted","Data":"7bad392e661e92d0f6bb77ca96ba14bf51e0f6fbda8f3f0a645cafc99ca69cda"} Jul 24 00:11:14.138091 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.137971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" event={"ID":"305fa5cc-79a6-4f1f-8d86-39bbf6633f1b","Type":"ContainerStarted","Data":"f5102477a90620db16f9a2349a61b4841d6e073cd1fff18b89c56c84c10722b2"} Jul 24 00:11:14.167464 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.167411 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zn62p" podStartSLOduration=2.186924474 podStartE2EDuration="4.16739279s" totalImagesPullingTime="1.980468316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:14.16680117 +0000 UTC m=+50.899436196" watchObservedRunningTime="2026-07-24 00:11:14.16739279 +0000 UTC m=+50.900028026" Jul 24 00:11:14.245335 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.245258 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nvdwf_1d88949d-b7a2-4105-9d66-793b76308f2b/node-ca/0.log" Jul 24 00:11:14.391643 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.391598 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:14.491485 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.491432 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:14.628916 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.628884 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:14.645391 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.645362 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jh5zk_7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee/serve-healthcheck-canary/0.log" Jul 24 00:11:14.846448 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:14.846400 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zn62p_305fa5cc-79a6-4f1f-8d86-39bbf6633f1b/migrator/0.log" Jul 24 00:11:15.044874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:15.044785 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zn62p_305fa5cc-79a6-4f1f-8d86-39bbf6633f1b/graceful-termination/0.log" Jul 24 00:11:15.216335 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:15.216298 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:15.216555 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:15.216444 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 00:11:15.216555 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:15.216533 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:19.216512672 +0000 UTC m=+55.949147679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : configmap "kube-rbac-proxy" not found Jul 24 00:11:15.216687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:15.216594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:15.216737 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:15.216728 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:11:15.216789 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:15.216781 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:19.216766938 +0000 UTC m=+55.949401955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : secret "insights-runtime-extractor-tls" not found Jul 24 00:11:15.248984 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:15.248955 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw_230ddb24-8a95-4058-b34a-25b6d01076d1/kube-storage-version-migrator-operator/0.log" Jul 24 00:11:15.526019 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:15.525966 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:16.528673 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:16.528641 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:16.528710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:16.528740 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:16.528772 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528846 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:32.528828372 +0000 UTC m=+69.261463378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : configmap references non-existent config key: service-ca.crt Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528904 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528909 2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528954 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs podName:5731834c-6e4d-474c-91d3-e92d41b96d00 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:32.528940604 +0000 UTC m=+69.261575629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs") pod "router-default-654cb6b4b5-ps8tt" (UID: "5731834c-6e4d-474c-91d3-e92d41b96d00") : secret "router-metrics-certs-default" not found Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528914 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528968 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls podName:6d8e7797-9ab7-4b76-a5cf-f6b16f548776 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:32.52896244 +0000 UTC m=+69.261597443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-bdrx8" (UID: "6d8e7797-9ab7-4b76-a5cf-f6b16f548776") : secret "cluster-monitoring-operator-tls" not found Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.528976 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cbc6d6f9c-8w96s: secret "image-registry-tls" not found Jul 24 00:11:16.529099 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.529007 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls podName:6a19ea5b-7f1e-4b45-ba88-e359708a673c nodeName:}" failed. No retries permitted until 2026-07-24 00:11:32.528993148 +0000 UTC m=+69.261628151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls") pod "image-registry-6cbc6d6f9c-8w96s" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c") : secret "image-registry-tls" not found Jul 24 00:11:16.630255 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:16.630216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:16.630424 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:16.630266 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:16.630424 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.630394 2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 00:11:16.630495 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.630423 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 00:11:16.630495 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.630452 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls podName:07fe13fd-0f87-4308-bd13-1b0fab1394bf nodeName:}" failed. No retries permitted until 2026-07-24 00:11:32.630437308 +0000 UTC m=+69.363072311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls") pod "cluster-samples-operator-6748489656-tgp7t" (UID: "07fe13fd-0f87-4308-bd13-1b0fab1394bf") : secret "samples-operator-tls" not found Jul 24 00:11:16.630495 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:16.630476 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls podName:0f4fe5bb-86b1-4155-a8cd-f3a151519316 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:32.630465258 +0000 UTC m=+69.363100261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls") pod "dns-default-nmfqx" (UID: "0f4fe5bb-86b1-4155-a8cd-f3a151519316") : secret "dns-default-metrics-tls" not found Jul 24 00:11:17.134692 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:17.134648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:17.134978 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:17.134792 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:11:17.134978 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:17.134853 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert podName:ca23905c-87a5-4884-ba50-74e081b46cea nodeName:}" failed. No retries permitted until 2026-07-24 00:11:25.13483932 +0000 UTC m=+61.867474326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-tgg5c" (UID: "ca23905c-87a5-4884-ba50-74e081b46cea") : secret "networking-console-plugin-cert" not found Jul 24 00:11:17.415306 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:17.415188 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:17.519165 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:17.519125 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:19.254571 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.254529 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:19.255193 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.254592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:19.255193 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:19.254684 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 00:11:19.255193 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:19.254750 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls podName:24722498-2095-48cc-941a-c82ef22e9ff1 nodeName:}" failed. No retries permitted until 2026-07-24 00:11:27.254734071 +0000 UTC m=+63.987369074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9zgc" (UID: "24722498-2095-48cc-941a-c82ef22e9ff1") : secret "insights-runtime-extractor-tls" not found Jul 24 00:11:19.255309 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.255195 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24722498-2095-48cc-941a-c82ef22e9ff1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:19.300807 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.300775 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:19.394932 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.394843 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:19.493103 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.493065 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:19.593922 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.593883 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:19.679347 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.679308 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:19.747241 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:19.747204 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:20.376943 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:20.376906 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:20.778654 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:20.778567 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:21.037113 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.037017 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:21.037113 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.037091 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:21.037459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.037446 2585 scope.go:117] "RemoveContainer" containerID="3d7c69d881ae7793e257892e5977eb7bbf245aea56952b88cfb469975f49dc77" Jul 24 00:11:21.158997 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.158973 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:11:21.159389 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.159374 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/1.log" Jul 24 00:11:21.159434 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.159412 2585 generic.go:356] "Generic (PLEG): container finished" podID="fd52790b-d34a-4c5b-8bf9-b1170f865154" containerID="820fdcb302ce8a655e9519b8f246b16bd0e80e475cbafd3956b75a20e0334120" exitCode=255 Jul 24 00:11:21.159502 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.159487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" event={"ID":"fd52790b-d34a-4c5b-8bf9-b1170f865154","Type":"ContainerDied","Data":"820fdcb302ce8a655e9519b8f246b16bd0e80e475cbafd3956b75a20e0334120"} Jul 24 00:11:21.159554 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.159542 2585 scope.go:117] "RemoveContainer" containerID="3d7c69d881ae7793e257892e5977eb7bbf245aea56952b88cfb469975f49dc77" Jul 24 00:11:21.159890 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.159851 2585 scope.go:117] "RemoveContainer" containerID="820fdcb302ce8a655e9519b8f246b16bd0e80e475cbafd3956b75a20e0334120" Jul 24 00:11:21.160065 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:21.160042 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-chwt8_openshift-console-operator(fd52790b-d34a-4c5b-8bf9-b1170f865154)\"" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" podUID="fd52790b-d34a-4c5b-8bf9-b1170f865154" Jul 24 00:11:21.182650 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.182621 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:21.599503 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.599467 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:21.984233 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:21.984145 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:22.163587 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:22.163560 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:11:22.385001 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:22.384971 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:22.780614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:22.780532 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:22.993643 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:22.993612 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:23.091463 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:23.091430 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:23.398162 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:23.398079 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:23.602843 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:23.602802 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.014766 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.014733 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.024660 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.024629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg7n2" Jul 24 00:11:24.164069 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.164030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.387186 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.387144 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.586603 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.586570 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.691443 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.691349 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.791141 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.791104 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:24.984186 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:24.984108 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:25.183276 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:25.183243 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:25.206337 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:25.206303 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:25.206478 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:25.206458 2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 00:11:25.206533 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:25.206527 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert podName:ca23905c-87a5-4884-ba50-74e081b46cea nodeName:}" failed. No retries permitted until 2026-07-24 00:11:41.206510289 +0000 UTC m=+77.939145292 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-tgg5c" (UID: "ca23905c-87a5-4884-ba50-74e081b46cea") : secret "networking-console-plugin-cert" not found Jul 24 00:11:25.577066 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:25.577030 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:25.650034 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:25.650000 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:25.979169 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:25.979078 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:26.381109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:26.381075 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:26.783168 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:26.783080 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:26.987327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:26.987296 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:27.324405 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.324362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:27.326666 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.326638 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24722498-2095-48cc-941a-c82ef22e9ff1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9zgc\" (UID: \"24722498-2095-48cc-941a-c82ef22e9ff1\") " pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:27.385046 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.384998 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:27.394712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.394679 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nsh42\"" Jul 24 00:11:27.402153 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.402127 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p9zgc" Jul 24 00:11:27.533580 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.533550 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p9zgc"] Jul 24 00:11:27.535981 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:27.535953 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24722498_2095_48cc_941a_c82ef22e9ff1.slice/crio-1937cd945527e9ba4ece6c14a9e7a0f34cd9cb41c381d5949f8102a40edff4a1 WatchSource:0}: Error finding container 1937cd945527e9ba4ece6c14a9e7a0f34cd9cb41c381d5949f8102a40edff4a1: Status 404 returned error can't find the container with id 1937cd945527e9ba4ece6c14a9e7a0f34cd9cb41c381d5949f8102a40edff4a1 Jul 24 00:11:27.979956 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:27.979912 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:28.184180 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.184139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9zgc" event={"ID":"24722498-2095-48cc-941a-c82ef22e9ff1","Type":"ContainerStarted","Data":"1937cd945527e9ba4ece6c14a9e7a0f34cd9cb41c381d5949f8102a40edff4a1"} Jul 24 00:11:28.536132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.536088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:11:28.536600 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.536206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:11:28.538853 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.538830 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 00:11:28.538853 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.538854 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 00:11:28.549653 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.549621 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833f0c21-b425-49fd-a639-2e2b235f1e7c-metrics-certs\") pod \"network-metrics-daemon-thh2r\" (UID: \"833f0c21-b425-49fd-a639-2e2b235f1e7c\") " pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:11:28.549784 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.549711 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3632857f-077a-4c1e-b064-dcb861c45095-original-pull-secret\") pod \"global-pull-secret-syncer-djmtj\" (UID: \"3632857f-077a-4c1e-b064-dcb861c45095\") " pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:11:28.590857 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.590819 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:28.637397 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.637347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:11:28.640278 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.640243 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbmr\" (UniqueName: \"kubernetes.io/projected/fd6a6183-624e-4420-99cd-df0beaa4ae40-kube-api-access-4cbmr\") pod \"network-check-target-rjqbq\" (UID: \"fd6a6183-624e-4420-99cd-df0beaa4ae40\") " pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:11:28.781763 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.781730 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:28.820217 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.820183 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvqg9\"" Jul 24 00:11:28.828573 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.828542 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-thh2r" Jul 24 00:11:28.832601 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.832571 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-79cjj\"" Jul 24 00:11:28.835832 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.835800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djmtj" Jul 24 00:11:28.840072 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.839929 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:11:28.979803 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.979702 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:28.996115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:28.996076 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-thh2r"] Jul 24 00:11:29.000625 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:29.000578 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833f0c21_b425_49fd_a639_2e2b235f1e7c.slice/crio-3ca61d8c37c566445dbf7b9fa728448933da9c0da2ceaa55ae24322175369384 WatchSource:0}: Error finding container 3ca61d8c37c566445dbf7b9fa728448933da9c0da2ceaa55ae24322175369384: Status 404 returned error can't find the container with id 3ca61d8c37c566445dbf7b9fa728448933da9c0da2ceaa55ae24322175369384 Jul 24 00:11:29.017093 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.017014 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rjqbq"] Jul 24 00:11:29.020083 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.020059 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-djmtj"] Jul 24 00:11:29.021391 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:29.021361 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6a6183_624e_4420_99cd_df0beaa4ae40.slice/crio-5f5898110646ad79c5863eed899921e7002307b911ccc0de75f31f9f7f891cc3 WatchSource:0}: Error finding container 5f5898110646ad79c5863eed899921e7002307b911ccc0de75f31f9f7f891cc3: Status 404 returned error can't find the container with id 5f5898110646ad79c5863eed899921e7002307b911ccc0de75f31f9f7f891cc3 Jul 24 00:11:29.022567 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:29.022546 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3632857f_077a_4c1e_b064_dcb861c45095.slice/crio-80db0ddb4979dcf4416b66b87ea6197d0946c6b08f6245200b827b52116939c9 WatchSource:0}: Error finding container 80db0ddb4979dcf4416b66b87ea6197d0946c6b08f6245200b827b52116939c9: Status 404 returned error can't find the container with id 80db0ddb4979dcf4416b66b87ea6197d0946c6b08f6245200b827b52116939c9 Jul 24 00:11:29.189466 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.189420 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rjqbq" event={"ID":"fd6a6183-624e-4420-99cd-df0beaa4ae40","Type":"ContainerStarted","Data":"dde2b00a58f4aa91c85ebd5d071806561a7764433ee64479329422a06e19bc53"} Jul 24 00:11:29.189667 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.189581 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:11:29.189667 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.189625 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rjqbq" event={"ID":"fd6a6183-624e-4420-99cd-df0beaa4ae40","Type":"ContainerStarted","Data":"5f5898110646ad79c5863eed899921e7002307b911ccc0de75f31f9f7f891cc3"} Jul 24 00:11:29.190927 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.190899 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-djmtj" event={"ID":"3632857f-077a-4c1e-b064-dcb861c45095","Type":"ContainerStarted","Data":"80db0ddb4979dcf4416b66b87ea6197d0946c6b08f6245200b827b52116939c9"} Jul 24 00:11:29.192312 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.192281 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thh2r" event={"ID":"833f0c21-b425-49fd-a639-2e2b235f1e7c","Type":"ContainerStarted","Data":"3ca61d8c37c566445dbf7b9fa728448933da9c0da2ceaa55ae24322175369384"} Jul 24 00:11:29.207794 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.207727 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rjqbq" podStartSLOduration=65.207706822 podStartE2EDuration="1m5.207706822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:29.207022863 +0000 UTC m=+65.939657888" watchObservedRunningTime="2026-07-24 00:11:29.207706822 +0000 UTC m=+65.940341847" Jul 24 00:11:29.389540 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.389454 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:29.838965 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.838922 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:29.978600 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:29.978563 2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" limit=1024 Jul 24 00:11:31.036877 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:31.036826 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:31.037366 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:31.036914 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:31.037435 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:31.037367 2585 scope.go:117] "RemoveContainer" containerID="820fdcb302ce8a655e9519b8f246b16bd0e80e475cbafd3956b75a20e0334120" Jul 24 00:11:31.037622 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:31.037596 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-chwt8_openshift-console-operator(fd52790b-d34a-4c5b-8bf9-b1170f865154)\"" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" podUID="fd52790b-d34a-4c5b-8bf9-b1170f865154" Jul 24 00:11:32.573817 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.573779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:32.574338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.573843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:32.574338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.573910 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:32.574338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.573965 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:32.574608 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.574583 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5731834c-6e4d-474c-91d3-e92d41b96d00-service-ca-bundle\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:32.576791 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.576763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"image-registry-6cbc6d6f9c-8w96s\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:32.576948 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.576855 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6d8e7797-9ab7-4b76-a5cf-f6b16f548776-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-bdrx8\" (UID: \"6d8e7797-9ab7-4b76-a5cf-f6b16f548776\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:32.577074 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.577051 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5731834c-6e4d-474c-91d3-e92d41b96d00-metrics-certs\") pod \"router-default-654cb6b4b5-ps8tt\" (UID: \"5731834c-6e4d-474c-91d3-e92d41b96d00\") " pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:32.674670 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.674627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:32.674896 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.674690 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:32.677364 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.677328 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0f4fe5bb-86b1-4155-a8cd-f3a151519316-metrics-tls\") pod \"dns-default-nmfqx\" (UID: \"0f4fe5bb-86b1-4155-a8cd-f3a151519316\") " pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:32.677493 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.677382 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/07fe13fd-0f87-4308-bd13-1b0fab1394bf-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tgp7t\" (UID: \"07fe13fd-0f87-4308-bd13-1b0fab1394bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:32.739327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.739295 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2gxlr\"" Jul 24 00:11:32.747048 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.747018 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:32.761986 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.761821 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5wmbl\"" Jul 24 00:11:32.768227 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.768179 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" Jul 24 00:11:32.787555 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.787525 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lfqbp\"" Jul 24 00:11:32.795033 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.795001 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:32.800942 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.800916 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j6lz4\"" Jul 24 00:11:32.808650 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.808617 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" Jul 24 00:11:32.889699 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.889661 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dcszn\"" Jul 24 00:11:32.897138 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:32.897110 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:34.828026 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:34.827667 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cbc6d6f9c-8w96s"] Jul 24 00:11:34.833353 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:34.833305 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a19ea5b_7f1e_4b45_ba88_e359708a673c.slice/crio-ad92b1eb0b9033ccfe9c7bef21169715c1b2929f9fc03006470b25e5e3e8de76 WatchSource:0}: Error finding container ad92b1eb0b9033ccfe9c7bef21169715c1b2929f9fc03006470b25e5e3e8de76: Status 404 returned error can't find the container with id ad92b1eb0b9033ccfe9c7bef21169715c1b2929f9fc03006470b25e5e3e8de76 Jul 24 00:11:34.851247 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:34.850998 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nmfqx"] Jul 24 00:11:34.861187 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:34.856625 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4fe5bb_86b1_4155_a8cd_f3a151519316.slice/crio-b149c19109977ae43f02f0220a75401ea622651eda5513318e9e7631a92599fb WatchSource:0}: Error finding container b149c19109977ae43f02f0220a75401ea622651eda5513318e9e7631a92599fb: Status 404 returned error can't find the container with id b149c19109977ae43f02f0220a75401ea622651eda5513318e9e7631a92599fb Jul 24 00:11:34.873792 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:34.873111 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-654cb6b4b5-ps8tt"] Jul 24 00:11:34.915329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:34.914715 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t"] Jul 24 00:11:35.068579 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.068491 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8"] Jul 24 00:11:35.073256 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:35.073214 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8e7797_9ab7_4b76_a5cf_f6b16f548776.slice/crio-f2582492562c4290e26b2f2eb21dceeffbfdc41055dcb80613b8c8e5e1651ed1 WatchSource:0}: Error finding container f2582492562c4290e26b2f2eb21dceeffbfdc41055dcb80613b8c8e5e1651ed1: Status 404 returned error can't find the container with id f2582492562c4290e26b2f2eb21dceeffbfdc41055dcb80613b8c8e5e1651ed1 Jul 24 00:11:35.217287 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.217186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9zgc" event={"ID":"24722498-2095-48cc-941a-c82ef22e9ff1","Type":"ContainerStarted","Data":"e857bf99ee3d268f659b78a17a7159cf9dc942ecd777db0d5276f4b1cd195274"} Jul 24 00:11:35.218431 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.218402 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" event={"ID":"07fe13fd-0f87-4308-bd13-1b0fab1394bf","Type":"ContainerStarted","Data":"17db1c2e6f4d5fe3bca4c076a4d064a8f15a4b17eed40be4183d6fcc2d7fa92d"} Jul 24 00:11:35.219972 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.219940 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" event={"ID":"5731834c-6e4d-474c-91d3-e92d41b96d00","Type":"ContainerStarted","Data":"9dc15b310914832fa8568532ef3f05f2cd56fb041adb406161aa1489e4115460"} Jul 24 00:11:35.220104 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.219986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" event={"ID":"5731834c-6e4d-474c-91d3-e92d41b96d00","Type":"ContainerStarted","Data":"cf99b082e5967599a9ad3ac6e80ea425462929853405c9dfd7b949a66713ff92"} Jul 24 00:11:35.221188 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.221167 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" event={"ID":"6d8e7797-9ab7-4b76-a5cf-f6b16f548776","Type":"ContainerStarted","Data":"f2582492562c4290e26b2f2eb21dceeffbfdc41055dcb80613b8c8e5e1651ed1"} Jul 24 00:11:35.222616 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.222597 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-djmtj" event={"ID":"3632857f-077a-4c1e-b064-dcb861c45095","Type":"ContainerStarted","Data":"b06de56707b37d16e4ad2b52512dd23c63b3a965263349e0b2050e65ed68443a"} Jul 24 00:11:35.224423 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.224398 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thh2r" event={"ID":"833f0c21-b425-49fd-a639-2e2b235f1e7c","Type":"ContainerStarted","Data":"f08bcf515a1771cc253d71ddc4be4131fd51c431cea2f8ac12ebbf9c4d45f9f4"} Jul 24 00:11:35.224515 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.224449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-thh2r" event={"ID":"833f0c21-b425-49fd-a639-2e2b235f1e7c","Type":"ContainerStarted","Data":"b80f50e45a5492ae6cf3f944fb095e25750bab4fcd3b08a7b0b98c16c24fae29"} Jul 24 00:11:35.225843 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.225824 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" event={"ID":"6a19ea5b-7f1e-4b45-ba88-e359708a673c","Type":"ContainerStarted","Data":"bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74"} Jul 24 00:11:35.225948 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.225874 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" event={"ID":"6a19ea5b-7f1e-4b45-ba88-e359708a673c","Type":"ContainerStarted","Data":"ad92b1eb0b9033ccfe9c7bef21169715c1b2929f9fc03006470b25e5e3e8de76"} Jul 24 00:11:35.225948 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.225904 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:35.226944 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.226926 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nmfqx" event={"ID":"0f4fe5bb-86b1-4155-a8cd-f3a151519316","Type":"ContainerStarted","Data":"b149c19109977ae43f02f0220a75401ea622651eda5513318e9e7631a92599fb"} Jul 24 00:11:35.254133 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.254065 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" podStartSLOduration=40.25404372 podStartE2EDuration="40.25404372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:35.253520166 +0000 UTC m=+71.986155193" watchObservedRunningTime="2026-07-24 00:11:35.25404372 +0000 UTC m=+71.986678750" Jul 24 00:11:35.281144 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.281075 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" podStartSLOduration=47.281053088 podStartE2EDuration="47.281053088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:35.279514526 +0000 UTC m=+72.012149576" watchObservedRunningTime="2026-07-24 00:11:35.281053088 +0000 UTC m=+72.013688113" Jul 24 00:11:35.299719 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.299635 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-djmtj" podStartSLOduration=64.726769789 podStartE2EDuration="1m10.299611845s" totalImagesPullingTime="5.572842056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:35.297543411 +0000 UTC m=+72.030178427" watchObservedRunningTime="2026-07-24 00:11:35.299611845 +0000 UTC m=+72.032246873" Jul 24 00:11:35.335887 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.335791 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-thh2r" podStartSLOduration=65.752261734 podStartE2EDuration="1m11.335771009s" totalImagesPullingTime="5.583509275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:35.335046949 +0000 UTC m=+72.067681980" watchObservedRunningTime="2026-07-24 00:11:35.335771009 +0000 UTC m=+72.068406034" Jul 24 00:11:35.802534 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.802499 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:35.802723 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:35.802619 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:36.233626 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:36.233571 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:36.235251 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:36.235061 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-654cb6b4b5-ps8tt" Jul 24 00:11:40.248976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.248935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" event={"ID":"6d8e7797-9ab7-4b76-a5cf-f6b16f548776","Type":"ContainerStarted","Data":"e2827d23234122124bf569e9dd74592431def6c0e93ddc60a56030fa7a5f9ef8"} Jul 24 00:11:40.250799 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.250769 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nmfqx" event={"ID":"0f4fe5bb-86b1-4155-a8cd-f3a151519316","Type":"ContainerStarted","Data":"df3cea055b75afef608c93ecd5217a6f85578f535b6944134f70336bcdf5b8e2"} Jul 24 00:11:40.252245 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.252186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" event={"ID":"07fe13fd-0f87-4308-bd13-1b0fab1394bf","Type":"ContainerStarted","Data":"5ee702c62efb8d787dbb4fa831079c83a98e8ece38cc7f9768280ccc9a0a44d8"} Jul 24 00:11:40.270370 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.270302 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-bdrx8" podStartSLOduration=40.345329609 podStartE2EDuration="45.270283195s" totalImagesPullingTime="4.924953586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:40.268034158 +0000 UTC m=+77.000669209" watchObservedRunningTime="2026-07-24 00:11:40.270283195 +0000 UTC m=+77.002918221" Jul 24 00:11:40.600970 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.600938 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k"] Jul 24 00:11:40.614308 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.614280 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:40.626657 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.626638 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 00:11:40.626797 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.626775 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4jkwx\"" Jul 24 00:11:40.638377 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.638346 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k"] Jul 24 00:11:40.749769 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.749736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8g56k\" (UID: \"e2d52d30-41dd-450d-8c10-0a4655c2455d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:40.850579 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:40.850529 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8g56k\" (UID: \"e2d52d30-41dd-450d-8c10-0a4655c2455d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:40.850820 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:40.850695 2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 24 00:11:40.850820 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:40.850766 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates podName:e2d52d30-41dd-450d-8c10-0a4655c2455d nodeName:}" failed. No retries permitted until 2026-07-24 00:11:41.350746669 +0000 UTC m=+78.083381672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-8g56k" (UID: "e2d52d30-41dd-450d-8c10-0a4655c2455d") : secret "prometheus-operator-admission-webhook-tls" not found Jul 24 00:11:41.253537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.253049 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:41.256095 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.256066 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ca23905c-87a5-4884-ba50-74e081b46cea-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-tgg5c\" (UID: \"ca23905c-87a5-4884-ba50-74e081b46cea\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:41.259087 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.259057 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nmfqx" event={"ID":"0f4fe5bb-86b1-4155-a8cd-f3a151519316","Type":"ContainerStarted","Data":"c48b92d577a5de42e735d3294ba1d08d5d7c16854df8c259efa097fe7f586f78"} Jul 24 00:11:41.259304 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.259249 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:41.261282 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.261255 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9zgc" event={"ID":"24722498-2095-48cc-941a-c82ef22e9ff1","Type":"ContainerStarted","Data":"aeeb139208a9b7ff78c26540ef9d052d800458ed4330bc8c2519edf975b79ea9"} Jul 24 00:11:41.264218 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.264194 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" event={"ID":"07fe13fd-0f87-4308-bd13-1b0fab1394bf","Type":"ContainerStarted","Data":"6963172a7f01b1c37ac5fd4fc154c920ec555872712d7b36aa8384c39a1131dd"} Jul 24 00:11:41.277735 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.277685 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nmfqx" podStartSLOduration=36.138103275 podStartE2EDuration="41.277669699s" totalImagesPullingTime="5.139566424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:41.276463169 +0000 UTC m=+78.009098196" watchObservedRunningTime="2026-07-24 00:11:41.277669699 +0000 UTC m=+78.010304724" Jul 24 00:11:41.293231 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.293179 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tgp7t" podStartSLOduration=41.254720029 podStartE2EDuration="46.293158648s" totalImagesPullingTime="5.038438619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:41.293080701 +0000 UTC m=+78.025715748" watchObservedRunningTime="2026-07-24 00:11:41.293158648 +0000 UTC m=+78.025793675" Jul 24 00:11:41.353622 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.353575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8g56k\" (UID: \"e2d52d30-41dd-450d-8c10-0a4655c2455d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:41.353809 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:41.353744 2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 24 00:11:41.353892 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:11:41.353810 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates podName:e2d52d30-41dd-450d-8c10-0a4655c2455d nodeName:}" failed. No retries permitted until 2026-07-24 00:11:42.353795204 +0000 UTC m=+79.086430207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-8g56k" (UID: "e2d52d30-41dd-450d-8c10-0a4655c2455d") : secret "prometheus-operator-admission-webhook-tls" not found Jul 24 00:11:41.553992 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.553969 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-sl59f\"" Jul 24 00:11:41.561509 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.561487 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" Jul 24 00:11:41.714700 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:41.714660 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c"] Jul 24 00:11:41.718583 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:41.718546 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca23905c_87a5_4884_ba50_74e081b46cea.slice/crio-0efcef6d8461e88f508ce7fad7deb29c943b470bf6212386eda3190d19dbbc34 WatchSource:0}: Error finding container 0efcef6d8461e88f508ce7fad7deb29c943b470bf6212386eda3190d19dbbc34: Status 404 returned error can't find the container with id 0efcef6d8461e88f508ce7fad7deb29c943b470bf6212386eda3190d19dbbc34 Jul 24 00:11:42.268823 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:42.268789 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" event={"ID":"ca23905c-87a5-4884-ba50-74e081b46cea","Type":"ContainerStarted","Data":"0efcef6d8461e88f508ce7fad7deb29c943b470bf6212386eda3190d19dbbc34"} Jul 24 00:11:42.362810 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:42.362770 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8g56k\" (UID: \"e2d52d30-41dd-450d-8c10-0a4655c2455d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:42.365643 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:42.365610 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e2d52d30-41dd-450d-8c10-0a4655c2455d-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8g56k\" (UID: \"e2d52d30-41dd-450d-8c10-0a4655c2455d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:42.423560 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:42.423514 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:42.793562 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:42.793530 2585 scope.go:117] "RemoveContainer" containerID="820fdcb302ce8a655e9519b8f246b16bd0e80e475cbafd3956b75a20e0334120" Jul 24 00:11:44.703817 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:44.703786 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k"] Jul 24 00:11:44.707510 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:44.707480 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d52d30_41dd_450d_8c10_0a4655c2455d.slice/crio-39b7663950c2780cbd184857363c317df27ffb147d76971dc525c0447b097432 WatchSource:0}: Error finding container 39b7663950c2780cbd184857363c317df27ffb147d76971dc525c0447b097432: Status 404 returned error can't find the container with id 39b7663950c2780cbd184857363c317df27ffb147d76971dc525c0447b097432 Jul 24 00:11:45.283974 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.283942 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" event={"ID":"ca23905c-87a5-4884-ba50-74e081b46cea","Type":"ContainerStarted","Data":"f3a8ec51ec3a0a5d90255eaf17c31e5b622de674d84dbf29375254a071202df7"} Jul 24 00:11:45.285315 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.285283 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" event={"ID":"e2d52d30-41dd-450d-8c10-0a4655c2455d","Type":"ContainerStarted","Data":"39b7663950c2780cbd184857363c317df27ffb147d76971dc525c0447b097432"} Jul 24 00:11:45.287314 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.287283 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9zgc" event={"ID":"24722498-2095-48cc-941a-c82ef22e9ff1","Type":"ContainerStarted","Data":"05fbe7805fce542d3a10fb62d8a56c9197be08238e0674554841279ce46d57e7"} Jul 24 00:11:45.289214 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.289194 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:11:45.289289 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.289257 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" event={"ID":"fd52790b-d34a-4c5b-8bf9-b1170f865154","Type":"ContainerStarted","Data":"03af8892f6c48a657346f8589ac081c82e43b7174829d53b953668c8396f5c99"} Jul 24 00:11:45.289507 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.289488 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:45.299751 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.299714 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-tgg5c" podStartSLOduration=33.444969186 podStartE2EDuration="36.299699357s" totalImagesPullingTime="2.854730171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:45.298511054 +0000 UTC m=+82.031146080" watchObservedRunningTime="2026-07-24 00:11:45.299699357 +0000 UTC m=+82.032334382" Jul 24 00:11:45.323578 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.323505 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" podStartSLOduration=42.632698403 podStartE2EDuration="50.323483074s" totalImagesPullingTime="7.690784671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:45.322723843 +0000 UTC m=+82.055358869" watchObservedRunningTime="2026-07-24 00:11:45.323483074 +0000 UTC m=+82.056118102" Jul 24 00:11:45.343058 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.343007 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p9zgc" podStartSLOduration=17.579079257 podStartE2EDuration="34.342988153s" totalImagesPullingTime="16.763908896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 00:11:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:45.341906176 +0000 UTC m=+82.074541202" watchObservedRunningTime="2026-07-24 00:11:45.342988153 +0000 UTC m=+82.075623172" Jul 24 00:11:45.654077 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:45.653958 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-chwt8" Jul 24 00:11:46.294896 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:46.294787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" event={"ID":"e2d52d30-41dd-450d-8c10-0a4655c2455d","Type":"ContainerStarted","Data":"4702945745a1ad898c6bdb6f77652706354442a41a84ae004d65b1f87245c233"} Jul 24 00:11:46.315279 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:46.315239 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" podStartSLOduration=5.011467601 podStartE2EDuration="6.315224083s" totalImagesPullingTime="1.303756482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:11:46.314074073 +0000 UTC m=+83.046709094" watchObservedRunningTime="2026-07-24 00:11:46.315224083 +0000 UTC m=+83.047859107" Jul 24 00:11:47.298659 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:47.298615 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:47.303326 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:47.303303 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8g56k" Jul 24 00:11:51.271616 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:51.271581 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nmfqx" Jul 24 00:11:56.237658 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:56.237630 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:11:58.080081 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:58.080043 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cbc6d6f9c-8w96s"] Jul 24 00:11:59.214697 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.214663 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9vlnq"] Jul 24 00:11:59.218045 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.218026 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.220972 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.220952 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 00:11:59.221109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.221020 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pmp2f\"" Jul 24 00:11:59.221109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.221020 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 00:11:59.221252 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.221237 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 00:11:59.221896 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.221881 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 00:11:59.283714 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283681 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-textfile\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.283928 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283736 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.283928 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283806 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4c85\" (UniqueName: \"kubernetes.io/projected/568bf52a-5f92-44ca-b3d4-278b6ae3d209-kube-api-access-h4c85\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.283928 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.283928 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283904 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-root\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.284097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283933 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568bf52a-5f92-44ca-b3d4-278b6ae3d209-metrics-client-ca\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.284097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.283962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-tls\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.284097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.284001 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-wtmp\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.284097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.284022 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-sys\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384442 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384459 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4c85\" (UniqueName: \"kubernetes.io/projected/568bf52a-5f92-44ca-b3d4-278b6ae3d209-kube-api-access-h4c85\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384488 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384512 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-root\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568bf52a-5f92-44ca-b3d4-278b6ae3d209-metrics-client-ca\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-tls\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384607 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-wtmp\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384634 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-sys\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384632 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-root\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-textfile\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384735 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-sys\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.384975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.384785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-wtmp\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.385112 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.385049 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-textfile\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.385375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.385347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568bf52a-5f92-44ca-b3d4-278b6ae3d209-metrics-client-ca\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.385421 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.385347 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.386811 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.386787 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.387007 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.386989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/568bf52a-5f92-44ca-b3d4-278b6ae3d209-node-exporter-tls\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.396323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.396298 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4c85\" (UniqueName: \"kubernetes.io/projected/568bf52a-5f92-44ca-b3d4-278b6ae3d209-kube-api-access-h4c85\") pod \"node-exporter-9vlnq\" (UID: \"568bf52a-5f92-44ca-b3d4-278b6ae3d209\") " pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.527448 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:11:59.527337 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vlnq" Jul 24 00:11:59.537903 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:11:59.537871 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568bf52a_5f92_44ca_b3d4_278b6ae3d209.slice/crio-6b63fabee62461bcf60bbe7b292aebed8fdcff4e0bd327ec7056e1b7257e565e WatchSource:0}: Error finding container 6b63fabee62461bcf60bbe7b292aebed8fdcff4e0bd327ec7056e1b7257e565e: Status 404 returned error can't find the container with id 6b63fabee62461bcf60bbe7b292aebed8fdcff4e0bd327ec7056e1b7257e565e Jul 24 00:12:00.200568 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:00.200535 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rjqbq" Jul 24 00:12:00.346002 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:00.345970 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vlnq" event={"ID":"568bf52a-5f92-44ca-b3d4-278b6ae3d209","Type":"ContainerStarted","Data":"6b63fabee62461bcf60bbe7b292aebed8fdcff4e0bd327ec7056e1b7257e565e"} Jul 24 00:12:01.351564 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:01.351530 2585 generic.go:356] "Generic (PLEG): container finished" podID="568bf52a-5f92-44ca-b3d4-278b6ae3d209" containerID="a7b9492051faa8841167e0a0a25c85c54d9aee7b0ef7db09f181d071e59f851d" exitCode=0 Jul 24 00:12:01.351989 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:01.351575 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vlnq" event={"ID":"568bf52a-5f92-44ca-b3d4-278b6ae3d209","Type":"ContainerDied","Data":"a7b9492051faa8841167e0a0a25c85c54d9aee7b0ef7db09f181d071e59f851d"} Jul 24 00:12:02.357329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:02.357297 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vlnq" event={"ID":"568bf52a-5f92-44ca-b3d4-278b6ae3d209","Type":"ContainerStarted","Data":"2d67a1a346f430a5a3c8f24b7ef43f5e3a7879d0d36a290a85bd6b0a50017684"} Jul 24 00:12:02.357923 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:02.357361 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vlnq" event={"ID":"568bf52a-5f92-44ca-b3d4-278b6ae3d209","Type":"ContainerStarted","Data":"89d514f9ff2b19997244482542cd9b0aacdff4e0b48ace0fe6df3328412ca7a0"} Jul 24 00:12:02.385352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:02.385299 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9vlnq" podStartSLOduration=1.819800504 podStartE2EDuration="3.385283882s" totalImagesPullingTime="1.500024424s" totalInitContainerRuntime="65.458954ms" isStatefulPod=true podCreationTimestamp="2026-07-24 00:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:12:02.379713931 +0000 UTC m=+99.112348956" watchObservedRunningTime="2026-07-24 00:12:02.385283882 +0000 UTC m=+99.117918906" Jul 24 00:12:23.098937 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.098897 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" podUID="6a19ea5b-7f1e-4b45-ba88-e359708a673c" containerName="registry" containerID="cri-o://bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74" gracePeriod=30 Jul 24 00:12:23.339725 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.339702 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:12:23.364943 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.364851 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-certificates\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365075 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.364948 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-installation-pull-secrets\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365075 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.364978 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a19ea5b-7f1e-4b45-ba88-e359708a673c-ca-trust-extracted\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365075 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365018 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-trusted-ca\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365075 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365065 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-image-registry-private-configuration\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365288 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365105 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-bound-sa-token\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365288 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365129 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365288 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365152 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsh4d\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-kube-api-access-xsh4d\") pod \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\" (UID: \"6a19ea5b-7f1e-4b45-ba88-e359708a673c\") " Jul 24 00:12:23.365444 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365359 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:12:23.365932 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.365909 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:12:23.368146 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.368098 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-kube-api-access-xsh4d" (OuterVolumeSpecName: "kube-api-access-xsh4d") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "kube-api-access-xsh4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:12:23.368146 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.368128 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:12:23.368295 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.368183 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:12:23.368357 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.368343 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:12:23.368627 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.368586 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:12:23.378338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.378305 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a19ea5b-7f1e-4b45-ba88-e359708a673c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6a19ea5b-7f1e-4b45-ba88-e359708a673c" (UID: "6a19ea5b-7f1e-4b45-ba88-e359708a673c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:12:23.434350 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.434312 2585 generic.go:356] "Generic (PLEG): container finished" podID="6a19ea5b-7f1e-4b45-ba88-e359708a673c" containerID="bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74" exitCode=0 Jul 24 00:12:23.434510 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.434355 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" Jul 24 00:12:23.434510 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.434371 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" event={"ID":"6a19ea5b-7f1e-4b45-ba88-e359708a673c","Type":"ContainerDied","Data":"bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74"} Jul 24 00:12:23.434510 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.434419 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cbc6d6f9c-8w96s" event={"ID":"6a19ea5b-7f1e-4b45-ba88-e359708a673c","Type":"ContainerDied","Data":"ad92b1eb0b9033ccfe9c7bef21169715c1b2929f9fc03006470b25e5e3e8de76"} Jul 24 00:12:23.434510 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.434443 2585 scope.go:117] "RemoveContainer" containerID="bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74" Jul 24 00:12:23.443048 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.443034 2585 scope.go:117] "RemoveContainer" containerID="bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74" Jul 24 00:12:23.443305 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:12:23.443284 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74\": container with ID starting with bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74 not found: ID does not exist" containerID="bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74" Jul 24 00:12:23.443349 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.443314 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74"} err="failed to get container status \"bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74\": rpc error: code = NotFound desc = could not find container \"bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74\": container with ID starting with bca423b12aad7abfe1df8123f1eda2701174035ba8dd5ff56fdc4f5014b6ed74 not found: ID does not exist" Jul 24 00:12:23.455704 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.455678 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cbc6d6f9c-8w96s"] Jul 24 00:12:23.459399 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.459378 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cbc6d6f9c-8w96s"] Jul 24 00:12:23.466264 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466248 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-trusted-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466267 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-image-registry-private-configuration\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466279 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-bound-sa-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466288 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466296 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsh4d\" (UniqueName: \"kubernetes.io/projected/6a19ea5b-7f1e-4b45-ba88-e359708a673c-kube-api-access-xsh4d\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466306 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6a19ea5b-7f1e-4b45-ba88-e359708a673c-registry-certificates\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466314 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6a19ea5b-7f1e-4b45-ba88-e359708a673c-installation-pull-secrets\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.466353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.466322 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6a19ea5b-7f1e-4b45-ba88-e359708a673c-ca-trust-extracted\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:12:23.798370 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:23.798335 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a19ea5b-7f1e-4b45-ba88-e359708a673c" path="/var/lib/kubelet/pods/6a19ea5b-7f1e-4b45-ba88-e359708a673c/volumes" Jul 24 00:12:25.443420 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:25.443328 2585 generic.go:356] "Generic (PLEG): container finished" podID="230ddb24-8a95-4058-b34a-25b6d01076d1" containerID="1d306cbf136f2d854b58231be5de4615ba5a1165e904dbba39d37729cdcb5a4d" exitCode=0 Jul 24 00:12:25.443889 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:25.443405 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" event={"ID":"230ddb24-8a95-4058-b34a-25b6d01076d1","Type":"ContainerDied","Data":"1d306cbf136f2d854b58231be5de4615ba5a1165e904dbba39d37729cdcb5a4d"} Jul 24 00:12:25.443889 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:25.443825 2585 scope.go:117] "RemoveContainer" containerID="1d306cbf136f2d854b58231be5de4615ba5a1165e904dbba39d37729cdcb5a4d" Jul 24 00:12:26.448389 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:26.448358 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw" event={"ID":"230ddb24-8a95-4058-b34a-25b6d01076d1","Type":"ContainerStarted","Data":"2d1bdba7f0648121f229855a29a036d3a446957db15374f40634f70343229df5"} Jul 24 00:12:31.471051 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:31.471017 2585 generic.go:356] "Generic (PLEG): container finished" podID="af80fcdb-8d4b-4585-93e5-032d9b1b8ac1" containerID="9801bea1ae7aedead4a9d43f1eb50d904e63a6d6e2259cdc762d0ba8d5b8af45" exitCode=0 Jul 24 00:12:31.471540 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:31.471092 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" event={"ID":"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1","Type":"ContainerDied","Data":"9801bea1ae7aedead4a9d43f1eb50d904e63a6d6e2259cdc762d0ba8d5b8af45"} Jul 24 00:12:31.471540 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:31.471456 2585 scope.go:117] "RemoveContainer" containerID="9801bea1ae7aedead4a9d43f1eb50d904e63a6d6e2259cdc762d0ba8d5b8af45" Jul 24 00:12:31.472736 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:31.472693 2585 generic.go:356] "Generic (PLEG): container finished" podID="0bb576d5-b2da-4ff1-9ca6-68f74da0ab02" containerID="e9b302612e80c488915c1c5abfa5d06eabd45f36adb18214e7a865e5a42a606e" exitCode=0 Jul 24 00:12:31.472736 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:31.472720 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qffj9" event={"ID":"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02","Type":"ContainerDied","Data":"e9b302612e80c488915c1c5abfa5d06eabd45f36adb18214e7a865e5a42a606e"} Jul 24 00:12:31.473024 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:31.472995 2585 scope.go:117] "RemoveContainer" containerID="e9b302612e80c488915c1c5abfa5d06eabd45f36adb18214e7a865e5a42a606e" Jul 24 00:12:32.478365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:32.478326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-rzb54" event={"ID":"af80fcdb-8d4b-4585-93e5-032d9b1b8ac1","Type":"ContainerStarted","Data":"17e08960ab87e8ed2e21ecc002056f9484e3f4dae68c72cca367275f635912db"} Jul 24 00:12:32.480046 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:12:32.480023 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-qffj9" event={"ID":"0bb576d5-b2da-4ff1-9ca6-68f74da0ab02","Type":"ContainerStarted","Data":"bf740ca4abae9241daa141ea8ea224c88bcb64632081a4298c6274c23c7a4092"} Jul 24 00:15:11.960248 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.960206 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2"] Jul 24 00:15:11.960952 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.960595 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a19ea5b-7f1e-4b45-ba88-e359708a673c" containerName="registry" Jul 24 00:15:11.960952 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.960611 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a19ea5b-7f1e-4b45-ba88-e359708a673c" containerName="registry" Jul 24 00:15:11.960952 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.960703 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a19ea5b-7f1e-4b45-ba88-e359708a673c" containerName="registry" Jul 24 00:15:11.963890 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.963849 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:11.966754 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.966731 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 00:15:11.967666 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.967641 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-52vls\"" Jul 24 00:15:11.967666 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.967657 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 00:15:11.973096 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:11.973069 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2"] Jul 24 00:15:12.048513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.048478 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.048513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.048519 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.048750 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.048557 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwfg\" (UniqueName: \"kubernetes.io/projected/752c8765-96e7-43e2-828c-c44b05681b1a-kube-api-access-8hwfg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.148947 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.148914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.148947 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.148948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.149190 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.148968 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwfg\" (UniqueName: \"kubernetes.io/projected/752c8765-96e7-43e2-828c-c44b05681b1a-kube-api-access-8hwfg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.149326 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.149306 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.149395 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.149377 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.158180 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.158153 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwfg\" (UniqueName: \"kubernetes.io/projected/752c8765-96e7-43e2-828c-c44b05681b1a-kube-api-access-8hwfg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.274200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.274109 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:12.401378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:12.401349 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2"] Jul 24 00:15:12.404012 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:15:12.403984 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752c8765_96e7_43e2_828c_c44b05681b1a.slice/crio-4ae74b9e304bd027efb3e019a7927aa5e778922e4d25744212b361950c21babb WatchSource:0}: Error finding container 4ae74b9e304bd027efb3e019a7927aa5e778922e4d25744212b361950c21babb: Status 404 returned error can't find the container with id 4ae74b9e304bd027efb3e019a7927aa5e778922e4d25744212b361950c21babb Jul 24 00:15:13.089854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:13.089821 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" event={"ID":"752c8765-96e7-43e2-828c-c44b05681b1a","Type":"ContainerStarted","Data":"4ae74b9e304bd027efb3e019a7927aa5e778922e4d25744212b361950c21babb"} Jul 24 00:15:18.110375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:18.110343 2585 generic.go:356] "Generic (PLEG): container finished" podID="752c8765-96e7-43e2-828c-c44b05681b1a" containerID="e13328eb255813d5f45cc80d17ff0f3691e8147c29cf95af2273cde272e42e7d" exitCode=0 Jul 24 00:15:18.110750 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:18.110424 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" event={"ID":"752c8765-96e7-43e2-828c-c44b05681b1a","Type":"ContainerDied","Data":"e13328eb255813d5f45cc80d17ff0f3691e8147c29cf95af2273cde272e42e7d"} Jul 24 00:15:21.124124 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:21.124090 2585 generic.go:356] "Generic (PLEG): container finished" podID="752c8765-96e7-43e2-828c-c44b05681b1a" containerID="80bb370bcdab478ba2a2103cb4f996c10aed187fd432817a1ce8ccec34c9ddbd" exitCode=0 Jul 24 00:15:21.124531 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:21.124182 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" event={"ID":"752c8765-96e7-43e2-828c-c44b05681b1a","Type":"ContainerDied","Data":"80bb370bcdab478ba2a2103cb4f996c10aed187fd432817a1ce8ccec34c9ddbd"} Jul 24 00:15:23.666369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:23.666330 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:15:23.669415 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:23.669389 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:15:23.682486 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:23.682458 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 00:15:28.153893 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:28.153835 2585 generic.go:356] "Generic (PLEG): container finished" podID="752c8765-96e7-43e2-828c-c44b05681b1a" containerID="f301447d9854b2b3243e282e508cedac26b60540f3d4a6428082c5f3614d8376" exitCode=0 Jul 24 00:15:28.154281 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:28.153891 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" event={"ID":"752c8765-96e7-43e2-828c-c44b05681b1a","Type":"ContainerDied","Data":"f301447d9854b2b3243e282e508cedac26b60540f3d4a6428082c5f3614d8376"} Jul 24 00:15:29.280630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.280606 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:29.382273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.382237 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-util\") pod \"752c8765-96e7-43e2-828c-c44b05681b1a\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " Jul 24 00:15:29.382435 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.382310 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hwfg\" (UniqueName: \"kubernetes.io/projected/752c8765-96e7-43e2-828c-c44b05681b1a-kube-api-access-8hwfg\") pod \"752c8765-96e7-43e2-828c-c44b05681b1a\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " Jul 24 00:15:29.382435 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.382331 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-bundle\") pod \"752c8765-96e7-43e2-828c-c44b05681b1a\" (UID: \"752c8765-96e7-43e2-828c-c44b05681b1a\") " Jul 24 00:15:29.383207 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.383169 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-bundle" (OuterVolumeSpecName: "bundle") pod "752c8765-96e7-43e2-828c-c44b05681b1a" (UID: "752c8765-96e7-43e2-828c-c44b05681b1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:29.384536 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.384509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752c8765-96e7-43e2-828c-c44b05681b1a-kube-api-access-8hwfg" (OuterVolumeSpecName: "kube-api-access-8hwfg") pod "752c8765-96e7-43e2-828c-c44b05681b1a" (UID: "752c8765-96e7-43e2-828c-c44b05681b1a"). InnerVolumeSpecName "kube-api-access-8hwfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:15:29.387545 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.387515 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-util" (OuterVolumeSpecName: "util") pod "752c8765-96e7-43e2-828c-c44b05681b1a" (UID: "752c8765-96e7-43e2-828c-c44b05681b1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:29.483243 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.483150 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hwfg\" (UniqueName: \"kubernetes.io/projected/752c8765-96e7-43e2-828c-c44b05681b1a-kube-api-access-8hwfg\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:29.483243 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.483189 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:29.483243 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:29.483202 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/752c8765-96e7-43e2-828c-c44b05681b1a-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:30.163484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:30.163406 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" Jul 24 00:15:30.163484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:30.163415 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pb8n2" event={"ID":"752c8765-96e7-43e2-828c-c44b05681b1a","Type":"ContainerDied","Data":"4ae74b9e304bd027efb3e019a7927aa5e778922e4d25744212b361950c21babb"} Jul 24 00:15:30.163484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:30.163458 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae74b9e304bd027efb3e019a7927aa5e778922e4d25744212b361950c21babb" Jul 24 00:15:35.001501 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001461 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj"] Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001750 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="util" Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001761 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="util" Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001772 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="pull" Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001778 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="pull" Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001796 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="extract" Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001802 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="extract" Jul 24 00:15:35.001901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.001850 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="752c8765-96e7-43e2-828c-c44b05681b1a" containerName="extract" Jul 24 00:15:35.003816 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.003800 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.015899 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.015874 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 00:15:35.016029 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.015881 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 00:15:35.024262 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.024236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce83c728-902e-4777-828c-d98fcd6adb72-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmjdj\" (UID: \"ce83c728-902e-4777-828c-d98fcd6adb72\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.024384 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.024281 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjgx\" (UniqueName: \"kubernetes.io/projected/ce83c728-902e-4777-828c-d98fcd6adb72-kube-api-access-6hjgx\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmjdj\" (UID: \"ce83c728-902e-4777-828c-d98fcd6adb72\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.026418 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.026398 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m8gg9\"" Jul 24 00:15:35.036584 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.036558 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj"] Jul 24 00:15:35.125611 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.125579 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce83c728-902e-4777-828c-d98fcd6adb72-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmjdj\" (UID: \"ce83c728-902e-4777-828c-d98fcd6adb72\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.125783 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.125626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjgx\" (UniqueName: \"kubernetes.io/projected/ce83c728-902e-4777-828c-d98fcd6adb72-kube-api-access-6hjgx\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmjdj\" (UID: \"ce83c728-902e-4777-828c-d98fcd6adb72\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.125996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.125978 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce83c728-902e-4777-828c-d98fcd6adb72-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmjdj\" (UID: \"ce83c728-902e-4777-828c-d98fcd6adb72\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.135158 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.135136 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjgx\" (UniqueName: \"kubernetes.io/projected/ce83c728-902e-4777-828c-d98fcd6adb72-kube-api-access-6hjgx\") pod \"cert-manager-operator-controller-manager-55c444bf86-bmjdj\" (UID: \"ce83c728-902e-4777-828c-d98fcd6adb72\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.312846 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.312813 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" Jul 24 00:15:35.446174 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.446059 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj"] Jul 24 00:15:35.448699 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:15:35.448673 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce83c728_902e_4777_828c_d98fcd6adb72.slice/crio-e0f05c3c231dd06939875cc9496752ecfc4ae8345b92ab855a75c0ff6d3c6c68 WatchSource:0}: Error finding container e0f05c3c231dd06939875cc9496752ecfc4ae8345b92ab855a75c0ff6d3c6c68: Status 404 returned error can't find the container with id e0f05c3c231dd06939875cc9496752ecfc4ae8345b92ab855a75c0ff6d3c6c68 Jul 24 00:15:35.451388 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:35.451373 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:15:36.188938 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:36.188897 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" event={"ID":"ce83c728-902e-4777-828c-d98fcd6adb72","Type":"ContainerStarted","Data":"e0f05c3c231dd06939875cc9496752ecfc4ae8345b92ab855a75c0ff6d3c6c68"} Jul 24 00:15:37.195065 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:37.195030 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" event={"ID":"ce83c728-902e-4777-828c-d98fcd6adb72","Type":"ContainerStarted","Data":"9c7f70818dcaddf8955cbb8eef852755b34fc1d1c11f2c480b0c36decbb74906"} Jul 24 00:15:37.218084 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:37.218031 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bmjdj" podStartSLOduration=1.655864923 podStartE2EDuration="3.218014801s" totalImagesPullingTime="1.562149878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:15:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:37.217572667 +0000 UTC m=+313.950207692" watchObservedRunningTime="2026-07-24 00:15:37.218014801 +0000 UTC m=+313.950649828" Jul 24 00:15:38.639765 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.639732 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l"] Jul 24 00:15:38.642082 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.642066 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.645338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.645314 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 00:15:38.645457 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.645377 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-52vls\"" Jul 24 00:15:38.646397 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.646380 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 00:15:38.653452 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.653431 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l"] Jul 24 00:15:38.754239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.754200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.754239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.754238 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.754462 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.754340 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql5l\" (UniqueName: \"kubernetes.io/projected/b3cbb5fe-d262-40c0-bc37-425d4f25e960-kube-api-access-tql5l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.855018 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.854981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.855018 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.855022 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.855207 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.855112 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql5l\" (UniqueName: \"kubernetes.io/projected/b3cbb5fe-d262-40c0-bc37-425d4f25e960-kube-api-access-tql5l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.855472 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.855451 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.855508 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.855483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.863469 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.863449 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql5l\" (UniqueName: \"kubernetes.io/projected/b3cbb5fe-d262-40c0-bc37-425d4f25e960-kube-api-access-tql5l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:38.951912 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:38.951801 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:39.087630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:39.087603 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l"] Jul 24 00:15:39.089900 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:15:39.089855 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cbb5fe_d262_40c0_bc37_425d4f25e960.slice/crio-baba4c70f7492509f63662db3ba56229eb17ff94cf8033c39b8f53bcd994e55c WatchSource:0}: Error finding container baba4c70f7492509f63662db3ba56229eb17ff94cf8033c39b8f53bcd994e55c: Status 404 returned error can't find the container with id baba4c70f7492509f63662db3ba56229eb17ff94cf8033c39b8f53bcd994e55c Jul 24 00:15:39.204239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:39.204165 2585 generic.go:356] "Generic (PLEG): container finished" podID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerID="7d60db20f22944f37d4b873d31b1ceec8e5dd24b48bb3ad12efbd08a67bbdbd4" exitCode=0 Jul 24 00:15:39.204239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:39.204204 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" event={"ID":"b3cbb5fe-d262-40c0-bc37-425d4f25e960","Type":"ContainerDied","Data":"7d60db20f22944f37d4b873d31b1ceec8e5dd24b48bb3ad12efbd08a67bbdbd4"} Jul 24 00:15:39.204420 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:39.204258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" event={"ID":"b3cbb5fe-d262-40c0-bc37-425d4f25e960","Type":"ContainerStarted","Data":"baba4c70f7492509f63662db3ba56229eb17ff94cf8033c39b8f53bcd994e55c"} Jul 24 00:15:41.215502 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.215413 2585 generic.go:356] "Generic (PLEG): container finished" podID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerID="0e6e1ef52055fabaf493158a5f7e5aaaf7ff570ec31c34e9b122338b6d90fb44" exitCode=0 Jul 24 00:15:41.215502 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.215462 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" event={"ID":"b3cbb5fe-d262-40c0-bc37-425d4f25e960","Type":"ContainerDied","Data":"0e6e1ef52055fabaf493158a5f7e5aaaf7ff570ec31c34e9b122338b6d90fb44"} Jul 24 00:15:41.569808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.569771 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9"] Jul 24 00:15:41.572920 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.572903 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.575840 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.575816 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 00:15:41.575968 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.575839 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qzb2s\"" Jul 24 00:15:41.576111 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.576098 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 00:15:41.583029 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.583005 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9"] Jul 24 00:15:41.680396 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.680359 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndx8\" (UniqueName: \"kubernetes.io/projected/cc62117c-5587-43ad-9745-a67842216b41-kube-api-access-cndx8\") pod \"cert-manager-cainjector-559cdd8c87-gq6q9\" (UID: \"cc62117c-5587-43ad-9745-a67842216b41\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.680396 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.680399 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc62117c-5587-43ad-9745-a67842216b41-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gq6q9\" (UID: \"cc62117c-5587-43ad-9745-a67842216b41\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.781754 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.781710 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndx8\" (UniqueName: \"kubernetes.io/projected/cc62117c-5587-43ad-9745-a67842216b41-kube-api-access-cndx8\") pod \"cert-manager-cainjector-559cdd8c87-gq6q9\" (UID: \"cc62117c-5587-43ad-9745-a67842216b41\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.781983 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.781759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc62117c-5587-43ad-9745-a67842216b41-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gq6q9\" (UID: \"cc62117c-5587-43ad-9745-a67842216b41\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.809701 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.809670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc62117c-5587-43ad-9745-a67842216b41-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gq6q9\" (UID: \"cc62117c-5587-43ad-9745-a67842216b41\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.809848 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.809720 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndx8\" (UniqueName: \"kubernetes.io/projected/cc62117c-5587-43ad-9745-a67842216b41-kube-api-access-cndx8\") pod \"cert-manager-cainjector-559cdd8c87-gq6q9\" (UID: \"cc62117c-5587-43ad-9745-a67842216b41\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:41.894851 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:41.894761 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" Jul 24 00:15:42.020659 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:42.020625 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9"] Jul 24 00:15:42.023406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:15:42.023372 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc62117c_5587_43ad_9745_a67842216b41.slice/crio-6453d98937f8b234400c90293c2ed8caede25bf0706b6bb9b61224a3d1576775 WatchSource:0}: Error finding container 6453d98937f8b234400c90293c2ed8caede25bf0706b6bb9b61224a3d1576775: Status 404 returned error can't find the container with id 6453d98937f8b234400c90293c2ed8caede25bf0706b6bb9b61224a3d1576775 Jul 24 00:15:42.220936 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:42.220828 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" event={"ID":"cc62117c-5587-43ad-9745-a67842216b41","Type":"ContainerStarted","Data":"6453d98937f8b234400c90293c2ed8caede25bf0706b6bb9b61224a3d1576775"} Jul 24 00:15:42.222821 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:42.222798 2585 generic.go:356] "Generic (PLEG): container finished" podID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerID="1b9fe4ce673f9dd9db86faa65ae2dc79bbf2abba01f50e45be6deb8fca48691e" exitCode=0 Jul 24 00:15:42.222948 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:42.222875 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" event={"ID":"b3cbb5fe-d262-40c0-bc37-425d4f25e960","Type":"ContainerDied","Data":"1b9fe4ce673f9dd9db86faa65ae2dc79bbf2abba01f50e45be6deb8fca48691e"} Jul 24 00:15:43.379520 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.379495 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:43.496904 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.496806 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-util\") pod \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " Jul 24 00:15:43.496904 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.496854 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tql5l\" (UniqueName: \"kubernetes.io/projected/b3cbb5fe-d262-40c0-bc37-425d4f25e960-kube-api-access-tql5l\") pod \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " Jul 24 00:15:43.497118 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.496951 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-bundle\") pod \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\" (UID: \"b3cbb5fe-d262-40c0-bc37-425d4f25e960\") " Jul 24 00:15:43.497378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.497332 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-bundle" (OuterVolumeSpecName: "bundle") pod "b3cbb5fe-d262-40c0-bc37-425d4f25e960" (UID: "b3cbb5fe-d262-40c0-bc37-425d4f25e960"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:43.499419 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.499392 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cbb5fe-d262-40c0-bc37-425d4f25e960-kube-api-access-tql5l" (OuterVolumeSpecName: "kube-api-access-tql5l") pod "b3cbb5fe-d262-40c0-bc37-425d4f25e960" (UID: "b3cbb5fe-d262-40c0-bc37-425d4f25e960"). InnerVolumeSpecName "kube-api-access-tql5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:15:43.503191 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.503146 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-util" (OuterVolumeSpecName: "util") pod "b3cbb5fe-d262-40c0-bc37-425d4f25e960" (UID: "b3cbb5fe-d262-40c0-bc37-425d4f25e960"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:43.598420 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.598384 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:43.598420 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.598413 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3cbb5fe-d262-40c0-bc37-425d4f25e960-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:43.598420 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:43.598422 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tql5l\" (UniqueName: \"kubernetes.io/projected/b3cbb5fe-d262-40c0-bc37-425d4f25e960-kube-api-access-tql5l\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:44.235054 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:44.234976 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" Jul 24 00:15:44.235208 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:44.234967 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f66v2l" event={"ID":"b3cbb5fe-d262-40c0-bc37-425d4f25e960","Type":"ContainerDied","Data":"baba4c70f7492509f63662db3ba56229eb17ff94cf8033c39b8f53bcd994e55c"} Jul 24 00:15:44.235208 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:44.235113 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baba4c70f7492509f63662db3ba56229eb17ff94cf8033c39b8f53bcd994e55c" Jul 24 00:15:45.240375 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:45.240335 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" event={"ID":"cc62117c-5587-43ad-9745-a67842216b41","Type":"ContainerStarted","Data":"45c7802f4c78c8a4e4814248a738a370f9609a88d7f6db317103bd699bec9ff8"} Jul 24 00:15:45.258673 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:45.258618 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gq6q9" podStartSLOduration=1.629046494 podStartE2EDuration="4.2586015s" totalImagesPullingTime="2.629555006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:45.25660177 +0000 UTC m=+321.989236797" watchObservedRunningTime="2026-07-24 00:15:45.2586015 +0000 UTC m=+321.991236525" Jul 24 00:15:48.393818 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.393783 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lvpfm"] Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394078 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="util" Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394089 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="util" Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394113 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="pull" Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394118 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="pull" Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394124 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="extract" Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394130 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="extract" Jul 24 00:15:48.394201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.394182 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cbb5fe-d262-40c0-bc37-425d4f25e960" containerName="extract" Jul 24 00:15:48.397017 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.396999 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.399568 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.399547 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c8nnm\"" Jul 24 00:15:48.409509 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.409486 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lvpfm"] Jul 24 00:15:48.442075 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.442033 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa18307e-ef98-4180-9923-ec22b17de00a-bound-sa-token\") pod \"cert-manager-55b4d954d5-lvpfm\" (UID: \"fa18307e-ef98-4180-9923-ec22b17de00a\") " pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.442222 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.442097 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zbr\" (UniqueName: \"kubernetes.io/projected/fa18307e-ef98-4180-9923-ec22b17de00a-kube-api-access-x8zbr\") pod \"cert-manager-55b4d954d5-lvpfm\" (UID: \"fa18307e-ef98-4180-9923-ec22b17de00a\") " pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.543397 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.543361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zbr\" (UniqueName: \"kubernetes.io/projected/fa18307e-ef98-4180-9923-ec22b17de00a-kube-api-access-x8zbr\") pod \"cert-manager-55b4d954d5-lvpfm\" (UID: \"fa18307e-ef98-4180-9923-ec22b17de00a\") " pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.543586 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.543424 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa18307e-ef98-4180-9923-ec22b17de00a-bound-sa-token\") pod \"cert-manager-55b4d954d5-lvpfm\" (UID: \"fa18307e-ef98-4180-9923-ec22b17de00a\") " pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.552986 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.552959 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa18307e-ef98-4180-9923-ec22b17de00a-bound-sa-token\") pod \"cert-manager-55b4d954d5-lvpfm\" (UID: \"fa18307e-ef98-4180-9923-ec22b17de00a\") " pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.553362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.553340 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zbr\" (UniqueName: \"kubernetes.io/projected/fa18307e-ef98-4180-9923-ec22b17de00a-kube-api-access-x8zbr\") pod \"cert-manager-55b4d954d5-lvpfm\" (UID: \"fa18307e-ef98-4180-9923-ec22b17de00a\") " pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.705456 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.705365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lvpfm" Jul 24 00:15:48.832449 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:48.832421 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lvpfm"] Jul 24 00:15:48.835039 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:15:48.835011 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa18307e_ef98_4180_9923_ec22b17de00a.slice/crio-d6399940477d5b6d2e3e58debe7bb09a6e8abde04650c1315d3a3925222082e0 WatchSource:0}: Error finding container d6399940477d5b6d2e3e58debe7bb09a6e8abde04650c1315d3a3925222082e0: Status 404 returned error can't find the container with id d6399940477d5b6d2e3e58debe7bb09a6e8abde04650c1315d3a3925222082e0 Jul 24 00:15:49.258748 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:49.258710 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lvpfm" event={"ID":"fa18307e-ef98-4180-9923-ec22b17de00a","Type":"ContainerStarted","Data":"ceec416494a1900c3abf8c9aa56f153de78eabb215afcf1c8ca937e1e0d4d3fb"} Jul 24 00:15:49.258952 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:49.258758 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lvpfm" event={"ID":"fa18307e-ef98-4180-9923-ec22b17de00a","Type":"ContainerStarted","Data":"d6399940477d5b6d2e3e58debe7bb09a6e8abde04650c1315d3a3925222082e0"} Jul 24 00:15:49.276536 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:49.276481 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lvpfm" podStartSLOduration=1.27646646 podStartE2EDuration="1.27646646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:15:49.275098889 +0000 UTC m=+326.007733968" watchObservedRunningTime="2026-07-24 00:15:49.27646646 +0000 UTC m=+326.009101484" Jul 24 00:15:55.116381 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.116346 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx"] Jul 24 00:15:55.123348 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.123328 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.126366 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.126342 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-52vls\"" Jul 24 00:15:55.126496 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.126371 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 00:15:55.127266 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.127245 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 00:15:55.129300 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.129268 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx"] Jul 24 00:15:55.196659 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.196620 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwvnx\" (UniqueName: \"kubernetes.io/projected/291cb65c-37ab-4312-97f5-3e9f77486dea-kube-api-access-xwvnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.196659 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.196663 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.196898 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.196741 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.297391 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.297350 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.297548 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.297406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwvnx\" (UniqueName: \"kubernetes.io/projected/291cb65c-37ab-4312-97f5-3e9f77486dea-kube-api-access-xwvnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.297548 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.297429 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.297718 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.297699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.297782 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.297759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.305982 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.305956 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwvnx\" (UniqueName: \"kubernetes.io/projected/291cb65c-37ab-4312-97f5-3e9f77486dea-kube-api-access-xwvnx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.434789 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.434702 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:55.564274 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:55.564242 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx"] Jul 24 00:15:55.568116 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:15:55.568081 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291cb65c_37ab_4312_97f5_3e9f77486dea.slice/crio-54e0f4e1aff2cb1e572b76846a1bf3fbe840ad6b8cfd8cb7075a985462a7c539 WatchSource:0}: Error finding container 54e0f4e1aff2cb1e572b76846a1bf3fbe840ad6b8cfd8cb7075a985462a7c539: Status 404 returned error can't find the container with id 54e0f4e1aff2cb1e572b76846a1bf3fbe840ad6b8cfd8cb7075a985462a7c539 Jul 24 00:15:56.287939 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:56.287898 2585 generic.go:356] "Generic (PLEG): container finished" podID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerID="9028cfaecbb78bedbeb16d2091173a1667a0e788bc4dede2b6f0a38df1e562c8" exitCode=0 Jul 24 00:15:56.288423 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:56.287992 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" event={"ID":"291cb65c-37ab-4312-97f5-3e9f77486dea","Type":"ContainerDied","Data":"9028cfaecbb78bedbeb16d2091173a1667a0e788bc4dede2b6f0a38df1e562c8"} Jul 24 00:15:56.288423 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:56.288039 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" event={"ID":"291cb65c-37ab-4312-97f5-3e9f77486dea","Type":"ContainerStarted","Data":"54e0f4e1aff2cb1e572b76846a1bf3fbe840ad6b8cfd8cb7075a985462a7c539"} Jul 24 00:15:57.294290 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:57.294198 2585 generic.go:356] "Generic (PLEG): container finished" podID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerID="b25c94587adbbc74f1bca0d0e5a5ef73489675acaa7b7ce2428963845da999a0" exitCode=0 Jul 24 00:15:57.294290 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:57.294278 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" event={"ID":"291cb65c-37ab-4312-97f5-3e9f77486dea","Type":"ContainerDied","Data":"b25c94587adbbc74f1bca0d0e5a5ef73489675acaa7b7ce2428963845da999a0"} Jul 24 00:15:58.301156 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:58.301120 2585 generic.go:356] "Generic (PLEG): container finished" podID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerID="1f7dbe82b1f0e620847514d7c95df9a291c3535bb6b0bfc1ac6dade0a3928369" exitCode=0 Jul 24 00:15:58.301538 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:58.301210 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" event={"ID":"291cb65c-37ab-4312-97f5-3e9f77486dea","Type":"ContainerDied","Data":"1f7dbe82b1f0e620847514d7c95df9a291c3535bb6b0bfc1ac6dade0a3928369"} Jul 24 00:15:59.453973 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.453938 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:15:59.529605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.529563 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-util\") pod \"291cb65c-37ab-4312-97f5-3e9f77486dea\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " Jul 24 00:15:59.529772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.529633 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwvnx\" (UniqueName: \"kubernetes.io/projected/291cb65c-37ab-4312-97f5-3e9f77486dea-kube-api-access-xwvnx\") pod \"291cb65c-37ab-4312-97f5-3e9f77486dea\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " Jul 24 00:15:59.529772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.529720 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-bundle\") pod \"291cb65c-37ab-4312-97f5-3e9f77486dea\" (UID: \"291cb65c-37ab-4312-97f5-3e9f77486dea\") " Jul 24 00:15:59.530482 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.530449 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-bundle" (OuterVolumeSpecName: "bundle") pod "291cb65c-37ab-4312-97f5-3e9f77486dea" (UID: "291cb65c-37ab-4312-97f5-3e9f77486dea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:59.531758 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.531727 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291cb65c-37ab-4312-97f5-3e9f77486dea-kube-api-access-xwvnx" (OuterVolumeSpecName: "kube-api-access-xwvnx") pod "291cb65c-37ab-4312-97f5-3e9f77486dea" (UID: "291cb65c-37ab-4312-97f5-3e9f77486dea"). InnerVolumeSpecName "kube-api-access-xwvnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:15:59.534831 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.534809 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-util" (OuterVolumeSpecName: "util") pod "291cb65c-37ab-4312-97f5-3e9f77486dea" (UID: "291cb65c-37ab-4312-97f5-3e9f77486dea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:15:59.630537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.630450 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:59.630537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.630483 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/291cb65c-37ab-4312-97f5-3e9f77486dea-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:15:59.630537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:15:59.630492 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwvnx\" (UniqueName: \"kubernetes.io/projected/291cb65c-37ab-4312-97f5-3e9f77486dea-kube-api-access-xwvnx\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:00.310761 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:00.310728 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" Jul 24 00:16:00.310761 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:00.310747 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afs78hx" event={"ID":"291cb65c-37ab-4312-97f5-3e9f77486dea","Type":"ContainerDied","Data":"54e0f4e1aff2cb1e572b76846a1bf3fbe840ad6b8cfd8cb7075a985462a7c539"} Jul 24 00:16:00.310987 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:00.310798 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54e0f4e1aff2cb1e572b76846a1bf3fbe840ad6b8cfd8cb7075a985462a7c539" Jul 24 00:16:11.698555 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698520 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc"] Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698830 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="extract" Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698840 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="extract" Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698850 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="pull" Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698855 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="pull" Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698892 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="util" Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698900 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="util" Jul 24 00:16:11.699055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.698950 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="291cb65c-37ab-4312-97f5-3e9f77486dea" containerName="extract" Jul 24 00:16:11.704446 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.704423 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.708445 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.708422 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 00:16:11.708581 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.708510 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-g5wcl\"" Jul 24 00:16:11.708901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.708884 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 00:16:11.709024 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.709006 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 00:16:11.709278 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.709262 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 00:16:11.719969 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.719947 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc"] Jul 24 00:16:11.829457 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.829420 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71b9799e-d866-49ce-93cc-dc513adae89c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.829638 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.829464 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/71b9799e-d866-49ce-93cc-dc513adae89c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.829638 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.829583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz8q\" (UniqueName: \"kubernetes.io/projected/71b9799e-d866-49ce-93cc-dc513adae89c-kube-api-access-zcz8q\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.829721 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.829647 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/71b9799e-d866-49ce-93cc-dc513adae89c-manifests\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.829721 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.829665 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71b9799e-d866-49ce-93cc-dc513adae89c-tmp\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.930762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.930723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz8q\" (UniqueName: \"kubernetes.io/projected/71b9799e-d866-49ce-93cc-dc513adae89c-kube-api-access-zcz8q\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.930970 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.930778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/71b9799e-d866-49ce-93cc-dc513adae89c-manifests\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.930970 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.930794 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71b9799e-d866-49ce-93cc-dc513adae89c-tmp\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.930970 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.930814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71b9799e-d866-49ce-93cc-dc513adae89c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.930970 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.930840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/71b9799e-d866-49ce-93cc-dc513adae89c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.931278 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.931242 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/71b9799e-d866-49ce-93cc-dc513adae89c-manifests\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.933141 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.933118 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71b9799e-d866-49ce-93cc-dc513adae89c-tmp\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.933444 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.933428 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/71b9799e-d866-49ce-93cc-dc513adae89c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.933491 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.933477 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71b9799e-d866-49ce-93cc-dc513adae89c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:11.939302 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:11.939277 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz8q\" (UniqueName: \"kubernetes.io/projected/71b9799e-d866-49ce-93cc-dc513adae89c-kube-api-access-zcz8q\") pod \"opendatahub-operator-controller-manager-7cdb55f47c-26nfc\" (UID: \"71b9799e-d866-49ce-93cc-dc513adae89c\") " pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:12.014584 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:12.014503 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:12.172172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:12.172147 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc"] Jul 24 00:16:12.174605 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:12.174575 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b9799e_d866_49ce_93cc_dc513adae89c.slice/crio-f975df0f9ef3befea7485eb934cff3067f9e8dcff6245fab3ab0a4a30e854460 WatchSource:0}: Error finding container f975df0f9ef3befea7485eb934cff3067f9e8dcff6245fab3ab0a4a30e854460: Status 404 returned error can't find the container with id f975df0f9ef3befea7485eb934cff3067f9e8dcff6245fab3ab0a4a30e854460 Jul 24 00:16:12.364200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:12.364164 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" event={"ID":"71b9799e-d866-49ce-93cc-dc513adae89c","Type":"ContainerStarted","Data":"f975df0f9ef3befea7485eb934cff3067f9e8dcff6245fab3ab0a4a30e854460"} Jul 24 00:16:15.381215 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:15.381175 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" event={"ID":"71b9799e-d866-49ce-93cc-dc513adae89c","Type":"ContainerStarted","Data":"12a6408a745faf6986cbf78d151d0b56d5d7d3bef7f3dec6fc5d7d82555988cd"} Jul 24 00:16:16.386690 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:16.386650 2585 generic.go:356] "Generic (PLEG): container finished" podID="71b9799e-d866-49ce-93cc-dc513adae89c" containerID="12a6408a745faf6986cbf78d151d0b56d5d7d3bef7f3dec6fc5d7d82555988cd" exitCode=0 Jul 24 00:16:16.387078 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:16.386739 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" event={"ID":"71b9799e-d866-49ce-93cc-dc513adae89c","Type":"ContainerDied","Data":"12a6408a745faf6986cbf78d151d0b56d5d7d3bef7f3dec6fc5d7d82555988cd"} Jul 24 00:16:17.395307 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:17.395260 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" event={"ID":"71b9799e-d866-49ce-93cc-dc513adae89c","Type":"ContainerStarted","Data":"bf6277fec6a2ad0b7f55dc8be08828dbf288dfcb480e01678f5bfab14c1dafa0"} Jul 24 00:16:17.395784 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:17.395409 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:17.422517 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:17.422451 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" podStartSLOduration=2.065463894 podStartE2EDuration="6.422428289s" totalImagesPullingTime="3.338529413s" totalInitContainerRuntime="1.018434982s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:17.42040025 +0000 UTC m=+354.153035289" watchObservedRunningTime="2026-07-24 00:16:17.422428289 +0000 UTC m=+354.155063314" Jul 24 00:16:28.402644 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:28.402611 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7cdb55f47c-26nfc" Jul 24 00:16:31.339852 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.339816 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5"] Jul 24 00:16:31.349660 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.349637 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.352725 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.352700 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 00:16:31.352934 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.352909 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 00:16:31.353542 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.353526 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fkk28\"" Jul 24 00:16:31.362590 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.362562 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5"] Jul 24 00:16:31.390387 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.390350 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxp8b\" (UniqueName: \"kubernetes.io/projected/070900fd-ba4a-4155-9f55-5d2a6c76ece0-kube-api-access-qxp8b\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.390541 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.390401 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070900fd-ba4a-4155-9f55-5d2a6c76ece0-tls-certs\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.390541 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.390468 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070900fd-ba4a-4155-9f55-5d2a6c76ece0-tmp\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.491394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.491362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070900fd-ba4a-4155-9f55-5d2a6c76ece0-tls-certs\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.491591 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.491414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070900fd-ba4a-4155-9f55-5d2a6c76ece0-tmp\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.491591 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.491489 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxp8b\" (UniqueName: \"kubernetes.io/projected/070900fd-ba4a-4155-9f55-5d2a6c76ece0-kube-api-access-qxp8b\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.493802 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.493774 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/070900fd-ba4a-4155-9f55-5d2a6c76ece0-tmp\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.494115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.494093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070900fd-ba4a-4155-9f55-5d2a6c76ece0-tls-certs\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.524554 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.524518 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxp8b\" (UniqueName: \"kubernetes.io/projected/070900fd-ba4a-4155-9f55-5d2a6c76ece0-kube-api-access-qxp8b\") pod \"kube-auth-proxy-57dc97f664-f5sh5\" (UID: \"070900fd-ba4a-4155-9f55-5d2a6c76ece0\") " pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.555693 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.555655 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-969d5"] Jul 24 00:16:31.559811 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.559787 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:31.563332 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.563310 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7zjh5\"" Jul 24 00:16:31.563458 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.563395 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 00:16:31.576267 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.576236 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-969d5"] Jul 24 00:16:31.592778 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.592682 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a9e45d-b7d4-4c7c-956f-2893d8727066-cert\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:31.592778 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.592733 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jp7\" (UniqueName: \"kubernetes.io/projected/93a9e45d-b7d4-4c7c-956f-2893d8727066-kube-api-access-q2jp7\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:31.659241 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.659208 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" Jul 24 00:16:31.693550 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.693517 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a9e45d-b7d4-4c7c-956f-2893d8727066-cert\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:31.693730 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.693576 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jp7\" (UniqueName: \"kubernetes.io/projected/93a9e45d-b7d4-4c7c-956f-2893d8727066-kube-api-access-q2jp7\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:31.693730 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:16:31.693699 2585 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 00:16:31.693849 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:16:31.693781 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a9e45d-b7d4-4c7c-956f-2893d8727066-cert podName:93a9e45d-b7d4-4c7c-956f-2893d8727066 nodeName:}" failed. No retries permitted until 2026-07-24 00:16:32.193757066 +0000 UTC m=+368.926392088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/93a9e45d-b7d4-4c7c-956f-2893d8727066-cert") pod "odh-model-controller-7998c67dc4-969d5" (UID: "93a9e45d-b7d4-4c7c-956f-2893d8727066") : secret "odh-model-controller-webhook-cert" not found Jul 24 00:16:31.706537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.706503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jp7\" (UniqueName: \"kubernetes.io/projected/93a9e45d-b7d4-4c7c-956f-2893d8727066-kube-api-access-q2jp7\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:31.813732 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:31.813711 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5"] Jul 24 00:16:31.816365 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:31.816333 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070900fd_ba4a_4155_9f55_5d2a6c76ece0.slice/crio-9b1a3c339ea67f48327e4e766cae56144d5da0152ffe7ca798cd0d07e4ddd31e WatchSource:0}: Error finding container 9b1a3c339ea67f48327e4e766cae56144d5da0152ffe7ca798cd0d07e4ddd31e: Status 404 returned error can't find the container with id 9b1a3c339ea67f48327e4e766cae56144d5da0152ffe7ca798cd0d07e4ddd31e Jul 24 00:16:32.197133 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.197097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a9e45d-b7d4-4c7c-956f-2893d8727066-cert\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:32.199562 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.199532 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a9e45d-b7d4-4c7c-956f-2893d8727066-cert\") pod \"odh-model-controller-7998c67dc4-969d5\" (UID: \"93a9e45d-b7d4-4c7c-956f-2893d8727066\") " pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:32.464490 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.464402 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" event={"ID":"070900fd-ba4a-4155-9f55-5d2a6c76ece0","Type":"ContainerStarted","Data":"9b1a3c339ea67f48327e4e766cae56144d5da0152ffe7ca798cd0d07e4ddd31e"} Jul 24 00:16:32.470572 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.470547 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:32.538505 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.538475 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kcmk5"] Jul 24 00:16:32.542339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.542314 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.545096 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.545068 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 00:16:32.545207 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.545101 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pg9cj\"" Jul 24 00:16:32.558692 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.558657 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kcmk5"] Jul 24 00:16:32.600285 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.600253 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b61fb86-63ef-44d6-87c6-27376f9c1a68-tls-certs\") pod \"model-serving-api-54d89bd5fc-kcmk5\" (UID: \"3b61fb86-63ef-44d6-87c6-27376f9c1a68\") " pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.600472 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.600384 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hsg2\" (UniqueName: \"kubernetes.io/projected/3b61fb86-63ef-44d6-87c6-27376f9c1a68-kube-api-access-6hsg2\") pod \"model-serving-api-54d89bd5fc-kcmk5\" (UID: \"3b61fb86-63ef-44d6-87c6-27376f9c1a68\") " pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.619952 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.619925 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-969d5"] Jul 24 00:16:32.623209 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:32.623167 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a9e45d_b7d4_4c7c_956f_2893d8727066.slice/crio-24ef35dcd3fc385cfbbcf5f7bcf820c685ba489e83fdb12e77ded57bb0ce690c WatchSource:0}: Error finding container 24ef35dcd3fc385cfbbcf5f7bcf820c685ba489e83fdb12e77ded57bb0ce690c: Status 404 returned error can't find the container with id 24ef35dcd3fc385cfbbcf5f7bcf820c685ba489e83fdb12e77ded57bb0ce690c Jul 24 00:16:32.701625 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.701577 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hsg2\" (UniqueName: \"kubernetes.io/projected/3b61fb86-63ef-44d6-87c6-27376f9c1a68-kube-api-access-6hsg2\") pod \"model-serving-api-54d89bd5fc-kcmk5\" (UID: \"3b61fb86-63ef-44d6-87c6-27376f9c1a68\") " pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.701625 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.701622 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b61fb86-63ef-44d6-87c6-27376f9c1a68-tls-certs\") pod \"model-serving-api-54d89bd5fc-kcmk5\" (UID: \"3b61fb86-63ef-44d6-87c6-27376f9c1a68\") " pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.704680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.704602 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b61fb86-63ef-44d6-87c6-27376f9c1a68-tls-certs\") pod \"model-serving-api-54d89bd5fc-kcmk5\" (UID: \"3b61fb86-63ef-44d6-87c6-27376f9c1a68\") " pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.717778 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.714933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hsg2\" (UniqueName: \"kubernetes.io/projected/3b61fb86-63ef-44d6-87c6-27376f9c1a68-kube-api-access-6hsg2\") pod \"model-serving-api-54d89bd5fc-kcmk5\" (UID: \"3b61fb86-63ef-44d6-87c6-27376f9c1a68\") " pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.769008 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.768975 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs"] Jul 24 00:16:32.772541 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.772515 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.777134 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.777022 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 00:16:32.777134 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.777035 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 00:16:32.777134 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.777021 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 00:16:32.777742 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.777616 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 00:16:32.777910 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.777744 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2jw8x\"" Jul 24 00:16:32.790580 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.790549 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs"] Jul 24 00:16:32.803022 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.802942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.803022 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.802988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.803306 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.803041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.803306 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.803145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.803306 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.803173 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbc57\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-kube-api-access-dbc57\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.803306 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.803220 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.803306 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.803251 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84997e2e-68fe-432a-8c8e-85dd8419d40f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.856259 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.856218 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904682 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904810 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbc57\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-kube-api-access-dbc57\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84997e2e-68fe-432a-8c8e-85dd8419d40f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.904962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.905855 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.905416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.908820 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.908603 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84997e2e-68fe-432a-8c8e-85dd8419d40f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.908820 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.908709 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.909048 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.908930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.909985 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.909915 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.924052 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.920120 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:32.924052 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:32.923066 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbc57\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-kube-api-access-dbc57\") pod \"istiod-openshift-gateway-7b7995d5cd-98hzs\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:33.044587 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:33.044559 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-kcmk5"] Jul 24 00:16:33.046925 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:33.046892 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b61fb86_63ef_44d6_87c6_27376f9c1a68.slice/crio-733992a17f7fc95922d09b8c90209ccb932a629dae2c3ed3fc47b4b20b9cbddd WatchSource:0}: Error finding container 733992a17f7fc95922d09b8c90209ccb932a629dae2c3ed3fc47b4b20b9cbddd: Status 404 returned error can't find the container with id 733992a17f7fc95922d09b8c90209ccb932a629dae2c3ed3fc47b4b20b9cbddd Jul 24 00:16:33.086616 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:33.086583 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:33.247098 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:33.247074 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs"] Jul 24 00:16:33.249782 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:33.249748 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84997e2e_68fe_432a_8c8e_85dd8419d40f.slice/crio-dedbd03bcb5e94b37afcd3eb271b23ee91213772dcd056ad66a24bc0df0e12be WatchSource:0}: Error finding container dedbd03bcb5e94b37afcd3eb271b23ee91213772dcd056ad66a24bc0df0e12be: Status 404 returned error can't find the container with id dedbd03bcb5e94b37afcd3eb271b23ee91213772dcd056ad66a24bc0df0e12be Jul 24 00:16:33.472327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:33.472289 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" event={"ID":"84997e2e-68fe-432a-8c8e-85dd8419d40f","Type":"ContainerStarted","Data":"dedbd03bcb5e94b37afcd3eb271b23ee91213772dcd056ad66a24bc0df0e12be"} Jul 24 00:16:33.474029 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:33.473997 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" event={"ID":"93a9e45d-b7d4-4c7c-956f-2893d8727066","Type":"ContainerStarted","Data":"24ef35dcd3fc385cfbbcf5f7bcf820c685ba489e83fdb12e77ded57bb0ce690c"} Jul 24 00:16:33.476163 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:33.476136 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" event={"ID":"3b61fb86-63ef-44d6-87c6-27376f9c1a68","Type":"ContainerStarted","Data":"733992a17f7fc95922d09b8c90209ccb932a629dae2c3ed3fc47b4b20b9cbddd"} Jul 24 00:16:37.825109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:37.825067 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:16:37.825394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:37.825149 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 00:16:38.509442 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.509399 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" event={"ID":"93a9e45d-b7d4-4c7c-956f-2893d8727066","Type":"ContainerStarted","Data":"221282ab42fbf242737e1e0cbf579104a021961efe29e5f214c257af8a494fde"} Jul 24 00:16:38.509629 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.509518 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:38.511160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.511131 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" event={"ID":"3b61fb86-63ef-44d6-87c6-27376f9c1a68","Type":"ContainerStarted","Data":"9645a1398e492cfa6c7e5b7663d8683edc6d08d6a19eb76249b365726dd8c89e"} Jul 24 00:16:38.511308 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.511211 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:38.512760 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.512729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" event={"ID":"070900fd-ba4a-4155-9f55-5d2a6c76ece0","Type":"ContainerStarted","Data":"c40e0a97acf3ec60ad291c921a33b421c91507716fbd14c5fb5cf9056413454f"} Jul 24 00:16:38.514338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.514317 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" event={"ID":"84997e2e-68fe-432a-8c8e-85dd8419d40f","Type":"ContainerStarted","Data":"464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd"} Jul 24 00:16:38.514532 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.514515 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:38.516284 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.516260 2585 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-98hzs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 00:16:38.516365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.516306 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" podUID="84997e2e-68fe-432a-8c8e-85dd8419d40f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 00:16:38.531534 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.531479 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" podStartSLOduration=2.382342147 podStartE2EDuration="7.531465154s" totalImagesPullingTime="5.149123007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:38.53002235 +0000 UTC m=+375.262657367" watchObservedRunningTime="2026-07-24 00:16:38.531465154 +0000 UTC m=+375.264100179" Jul 24 00:16:38.554280 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.554222 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-57dc97f664-f5sh5" podStartSLOduration=2.520861968 podStartE2EDuration="7.554203447s" totalImagesPullingTime="5.033341479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:38.552376323 +0000 UTC m=+375.285011359" watchObservedRunningTime="2026-07-24 00:16:38.554203447 +0000 UTC m=+375.286838471" Jul 24 00:16:38.578484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.578425 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" podStartSLOduration=2.005587113 podStartE2EDuration="6.578403067s" totalImagesPullingTime="4.572815954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:38.57657081 +0000 UTC m=+375.309205831" watchObservedRunningTime="2026-07-24 00:16:38.578403067 +0000 UTC m=+375.311038091" Jul 24 00:16:38.616578 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:38.616508 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" podStartSLOduration=1.900675615 podStartE2EDuration="6.61648371s" totalImagesPullingTime="4.715808095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:38.613249751 +0000 UTC m=+375.345884781" watchObservedRunningTime="2026-07-24 00:16:38.61648371 +0000 UTC m=+375.349118736" Jul 24 00:16:39.523584 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:39.523539 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:44.153467 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.153432 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg"] Jul 24 00:16:44.156844 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.156828 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.164905 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.164886 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 00:16:44.171291 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.171268 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9gxqj\"" Jul 24 00:16:44.179079 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.179056 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92"] Jul 24 00:16:44.183002 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.182935 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.185937 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.185914 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg"] Jul 24 00:16:44.188814 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.188793 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-l8qm4\"" Jul 24 00:16:44.188966 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.188939 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 00:16:44.200971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.200945 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92"] Jul 24 00:16:44.208441 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.208410 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6g4p\" (UniqueName: \"kubernetes.io/projected/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-kube-api-access-r6g4p\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.208575 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.208447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-cert\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.309529 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.309491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6g4p\" (UniqueName: \"kubernetes.io/projected/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-kube-api-access-r6g4p\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.309529 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.309530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-cert\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.309764 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:16:44.309640 2585 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 24 00:16:44.309764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.309677 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68d247fa-5dfc-4592-b64e-21791ab80e24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tjs92\" (UID: \"68d247fa-5dfc-4592-b64e-21791ab80e24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.309764 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:16:44.309705 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-cert podName:9e0454cd-f302-4c17-a44a-1ab0c3afd4ed nodeName:}" failed. No retries permitted until 2026-07-24 00:16:44.809683133 +0000 UTC m=+381.542318136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-cert") pod "kserve-controller-manager-d6f4b65b8-kwnvg" (UID: "9e0454cd-f302-4c17-a44a-1ab0c3afd4ed") : secret "kserve-webhook-server-cert" not found Jul 24 00:16:44.309764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.309749 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vd4\" (UniqueName: \"kubernetes.io/projected/68d247fa-5dfc-4592-b64e-21791ab80e24-kube-api-access-d5vd4\") pod \"llmisvc-controller-manager-777c6bd7d5-tjs92\" (UID: \"68d247fa-5dfc-4592-b64e-21791ab80e24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.342587 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.342554 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6g4p\" (UniqueName: \"kubernetes.io/projected/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-kube-api-access-r6g4p\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.411211 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.411110 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68d247fa-5dfc-4592-b64e-21791ab80e24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tjs92\" (UID: \"68d247fa-5dfc-4592-b64e-21791ab80e24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.411211 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.411153 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vd4\" (UniqueName: \"kubernetes.io/projected/68d247fa-5dfc-4592-b64e-21791ab80e24-kube-api-access-d5vd4\") pod \"llmisvc-controller-manager-777c6bd7d5-tjs92\" (UID: \"68d247fa-5dfc-4592-b64e-21791ab80e24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.413798 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.413761 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68d247fa-5dfc-4592-b64e-21791ab80e24-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tjs92\" (UID: \"68d247fa-5dfc-4592-b64e-21791ab80e24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.444484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.444458 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vd4\" (UniqueName: \"kubernetes.io/projected/68d247fa-5dfc-4592-b64e-21791ab80e24-kube-api-access-d5vd4\") pod \"llmisvc-controller-manager-777c6bd7d5-tjs92\" (UID: \"68d247fa-5dfc-4592-b64e-21791ab80e24\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.495622 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.495586 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:44.659144 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.659047 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92"] Jul 24 00:16:44.661427 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:44.661366 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68d247fa_5dfc_4592_b64e_21791ab80e24.slice/crio-2b03284476e290f32166f96937ec3f1aea0760a1c0270ad754632481fa4b449e WatchSource:0}: Error finding container 2b03284476e290f32166f96937ec3f1aea0760a1c0270ad754632481fa4b449e: Status 404 returned error can't find the container with id 2b03284476e290f32166f96937ec3f1aea0760a1c0270ad754632481fa4b449e Jul 24 00:16:44.815563 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.815525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-cert\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:44.817899 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:44.817877 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9e0454cd-f302-4c17-a44a-1ab0c3afd4ed-cert\") pod \"kserve-controller-manager-d6f4b65b8-kwnvg\" (UID: \"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:45.067353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:45.067321 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:45.229385 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:45.229353 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg"] Jul 24 00:16:45.231694 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:16:45.231669 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0454cd_f302_4c17_a44a_1ab0c3afd4ed.slice/crio-6c6ecb23142dc12240ad29e26b086fe5f369f1c79d95f55349d215b46c5c43ef WatchSource:0}: Error finding container 6c6ecb23142dc12240ad29e26b086fe5f369f1c79d95f55349d215b46c5c43ef: Status 404 returned error can't find the container with id 6c6ecb23142dc12240ad29e26b086fe5f369f1c79d95f55349d215b46c5c43ef Jul 24 00:16:45.551595 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:45.551514 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" event={"ID":"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed","Type":"ContainerStarted","Data":"6c6ecb23142dc12240ad29e26b086fe5f369f1c79d95f55349d215b46c5c43ef"} Jul 24 00:16:45.552739 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:45.552708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" event={"ID":"68d247fa-5dfc-4592-b64e-21791ab80e24","Type":"ContainerStarted","Data":"2b03284476e290f32166f96937ec3f1aea0760a1c0270ad754632481fa4b449e"} Jul 24 00:16:47.202834 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:47.202803 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs"] Jul 24 00:16:47.203310 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:47.203039 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" podUID="84997e2e-68fe-432a-8c8e-85dd8419d40f" containerName="discovery" containerID="cri-o://464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd" gracePeriod=30 Jul 24 00:16:48.336589 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.336562 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:48.450289 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450259 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84997e2e-68fe-432a-8c8e-85dd8419d40f-local-certs\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.450449 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450336 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-dns-cert\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.450449 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450367 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-kubeconfig\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.450449 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450411 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-token\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.450614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450499 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-cacerts\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.450614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450559 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-ca-configmap\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.450614 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.450591 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbc57\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-kube-api-access-dbc57\") pod \"84997e2e-68fe-432a-8c8e-85dd8419d40f\" (UID: \"84997e2e-68fe-432a-8c8e-85dd8419d40f\") " Jul 24 00:16:48.451115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.451025 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:16:48.452980 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.452950 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:16:48.453200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.453150 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-token" (OuterVolumeSpecName: "istio-token") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:16:48.453370 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.453338 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-cacerts" (OuterVolumeSpecName: "cacerts") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:16:48.453460 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.453357 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84997e2e-68fe-432a-8c8e-85dd8419d40f-local-certs" (OuterVolumeSpecName: "local-certs") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:16:48.453460 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.453396 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-kube-api-access-dbc57" (OuterVolumeSpecName: "kube-api-access-dbc57") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "kube-api-access-dbc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:16:48.453460 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.453422 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "84997e2e-68fe-432a-8c8e-85dd8419d40f" (UID: "84997e2e-68fe-432a-8c8e-85dd8419d40f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:16:48.552039 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.551999 2585 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/84997e2e-68fe-432a-8c8e-85dd8419d40f-local-certs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.552039 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.552028 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-dns-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.552039 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.552040 2585 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-kubeconfig\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.552039 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.552048 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-token\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.552314 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.552059 2585 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/84997e2e-68fe-432a-8c8e-85dd8419d40f-cacerts\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.552314 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.552067 2585 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/84997e2e-68fe-432a-8c8e-85dd8419d40f-istio-csr-ca-configmap\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.552314 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.552076 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbc57\" (UniqueName: \"kubernetes.io/projected/84997e2e-68fe-432a-8c8e-85dd8419d40f-kube-api-access-dbc57\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:16:48.578382 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.578291 2585 generic.go:356] "Generic (PLEG): container finished" podID="84997e2e-68fe-432a-8c8e-85dd8419d40f" containerID="464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd" exitCode=0 Jul 24 00:16:48.578382 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.578342 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" Jul 24 00:16:48.578382 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.578357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" event={"ID":"84997e2e-68fe-432a-8c8e-85dd8419d40f","Type":"ContainerDied","Data":"464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd"} Jul 24 00:16:48.578646 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.578409 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs" event={"ID":"84997e2e-68fe-432a-8c8e-85dd8419d40f","Type":"ContainerDied","Data":"dedbd03bcb5e94b37afcd3eb271b23ee91213772dcd056ad66a24bc0df0e12be"} Jul 24 00:16:48.578646 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.578438 2585 scope.go:117] "RemoveContainer" containerID="464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd" Jul 24 00:16:48.580090 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.580063 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" event={"ID":"9e0454cd-f302-4c17-a44a-1ab0c3afd4ed","Type":"ContainerStarted","Data":"759bc3ca90d944b5a2405d2eef1d06f206228852bf4b2ba4f6ad69d9a23b1a8e"} Jul 24 00:16:48.580244 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.580121 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:16:48.581744 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.581722 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" event={"ID":"68d247fa-5dfc-4592-b64e-21791ab80e24","Type":"ContainerStarted","Data":"90b826f16ed66dabb86791c923a2240f32530be96c76900a6217bb917ae9e456"} Jul 24 00:16:48.581842 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.581812 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:16:48.588781 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.588764 2585 scope.go:117] "RemoveContainer" containerID="464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd" Jul 24 00:16:48.589077 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:16:48.589060 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd\": container with ID starting with 464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd not found: ID does not exist" containerID="464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd" Jul 24 00:16:48.589132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.589084 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd"} err="failed to get container status \"464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd\": rpc error: code = NotFound desc = could not find container \"464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd\": container with ID starting with 464bd7e465254c8bae30d147e6627cc64967458d0d4a9b852bba15ef631fe6bd not found: ID does not exist" Jul 24 00:16:48.634933 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.634855 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" podStartSLOduration=1.512807584 podStartE2EDuration="4.634828771s" totalImagesPullingTime="3.122021187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:48.633032756 +0000 UTC m=+385.365667794" watchObservedRunningTime="2026-07-24 00:16:48.634828771 +0000 UTC m=+385.367463832" Jul 24 00:16:48.664313 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.663547 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" podStartSLOduration=0.974855277 podStartE2EDuration="4.663525335s" totalImagesPullingTime="3.688670058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:16:48.66285542 +0000 UTC m=+385.395490467" watchObservedRunningTime="2026-07-24 00:16:48.663525335 +0000 UTC m=+385.396160361" Jul 24 00:16:48.677692 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.677660 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs"] Jul 24 00:16:48.681161 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:48.681135 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-98hzs"] Jul 24 00:16:49.524972 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:49.524941 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-969d5" Jul 24 00:16:49.526882 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:49.526832 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-kcmk5" Jul 24 00:16:49.799515 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:16:49.799426 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84997e2e-68fe-432a-8c8e-85dd8419d40f" path="/var/lib/kubelet/pods/84997e2e-68fe-432a-8c8e-85dd8419d40f/volumes" Jul 24 00:17:19.591854 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:19.591816 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tjs92" Jul 24 00:17:19.594812 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:19.594781 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kwnvg" Jul 24 00:17:32.482024 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.481987 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bkw6"] Jul 24 00:17:32.482379 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.482308 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84997e2e-68fe-432a-8c8e-85dd8419d40f" containerName="discovery" Jul 24 00:17:32.482379 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.482320 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="84997e2e-68fe-432a-8c8e-85dd8419d40f" containerName="discovery" Jul 24 00:17:32.482379 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.482378 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="84997e2e-68fe-432a-8c8e-85dd8419d40f" containerName="discovery" Jul 24 00:17:32.493702 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.493636 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bkw6"] Jul 24 00:17:32.493702 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.493687 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:32.496213 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.496185 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 00:17:32.496339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.496231 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-46nsk\"" Jul 24 00:17:32.497390 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.497346 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 00:17:32.516261 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.516235 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrn6r\" (UniqueName: \"kubernetes.io/projected/b28035e8-9696-43e6-9052-a144e47c0a2d-kube-api-access-vrn6r\") pod \"kuadrant-operator-catalog-6bkw6\" (UID: \"b28035e8-9696-43e6-9052-a144e47c0a2d\") " pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:32.617642 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.617609 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrn6r\" (UniqueName: \"kubernetes.io/projected/b28035e8-9696-43e6-9052-a144e47c0a2d-kube-api-access-vrn6r\") pod \"kuadrant-operator-catalog-6bkw6\" (UID: \"b28035e8-9696-43e6-9052-a144e47c0a2d\") " pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:32.625359 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.625333 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrn6r\" (UniqueName: \"kubernetes.io/projected/b28035e8-9696-43e6-9052-a144e47c0a2d-kube-api-access-vrn6r\") pod \"kuadrant-operator-catalog-6bkw6\" (UID: \"b28035e8-9696-43e6-9052-a144e47c0a2d\") " pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:32.803819 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.803799 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:32.861267 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.861197 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bkw6"] Jul 24 00:17:32.931343 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:32.931318 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bkw6"] Jul 24 00:17:32.933778 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:17:32.933746 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28035e8_9696_43e6_9052_a144e47c0a2d.slice/crio-14839047fc08d8a2755bf72ed9dbd49bf62a74a6c3c746b2c649c2c02e8af38b WatchSource:0}: Error finding container 14839047fc08d8a2755bf72ed9dbd49bf62a74a6c3c746b2c649c2c02e8af38b: Status 404 returned error can't find the container with id 14839047fc08d8a2755bf72ed9dbd49bf62a74a6c3c746b2c649c2c02e8af38b Jul 24 00:17:33.070469 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.070391 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgfqk"] Jul 24 00:17:33.073523 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.073504 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:33.080906 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.080840 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgfqk"] Jul 24 00:17:33.122153 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.122126 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxtr\" (UniqueName: \"kubernetes.io/projected/fdc3ae66-934f-428d-8916-3ad1a64463db-kube-api-access-vcxtr\") pod \"kuadrant-operator-catalog-fgfqk\" (UID: \"fdc3ae66-934f-428d-8916-3ad1a64463db\") " pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:33.223287 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.223252 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxtr\" (UniqueName: \"kubernetes.io/projected/fdc3ae66-934f-428d-8916-3ad1a64463db-kube-api-access-vcxtr\") pod \"kuadrant-operator-catalog-fgfqk\" (UID: \"fdc3ae66-934f-428d-8916-3ad1a64463db\") " pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:33.230713 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.230681 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxtr\" (UniqueName: \"kubernetes.io/projected/fdc3ae66-934f-428d-8916-3ad1a64463db-kube-api-access-vcxtr\") pod \"kuadrant-operator-catalog-fgfqk\" (UID: \"fdc3ae66-934f-428d-8916-3ad1a64463db\") " pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:33.384329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.384233 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:33.517958 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.517931 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fgfqk"] Jul 24 00:17:33.544905 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:17:33.544875 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc3ae66_934f_428d_8916_3ad1a64463db.slice/crio-01afc3a300910796d03e42a6d00abdbb86607d22036f6a13029519dd5f613510 WatchSource:0}: Error finding container 01afc3a300910796d03e42a6d00abdbb86607d22036f6a13029519dd5f613510: Status 404 returned error can't find the container with id 01afc3a300910796d03e42a6d00abdbb86607d22036f6a13029519dd5f613510 Jul 24 00:17:33.803440 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.803410 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" event={"ID":"b28035e8-9696-43e6-9052-a144e47c0a2d","Type":"ContainerStarted","Data":"14839047fc08d8a2755bf72ed9dbd49bf62a74a6c3c746b2c649c2c02e8af38b"} Jul 24 00:17:33.804840 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:33.804808 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" event={"ID":"fdc3ae66-934f-428d-8916-3ad1a64463db","Type":"ContainerStarted","Data":"01afc3a300910796d03e42a6d00abdbb86607d22036f6a13029519dd5f613510"} Jul 24 00:17:35.816199 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:35.816164 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" event={"ID":"fdc3ae66-934f-428d-8916-3ad1a64463db","Type":"ContainerStarted","Data":"501aeb8f55abc62fe12218f5dc69c9b2a9a77bccb15d9811d1d1e99876e8b5cb"} Jul 24 00:17:35.817615 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:35.817589 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" event={"ID":"b28035e8-9696-43e6-9052-a144e47c0a2d","Type":"ContainerStarted","Data":"07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c"} Jul 24 00:17:35.817615 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:35.817604 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" podUID="b28035e8-9696-43e6-9052-a144e47c0a2d" containerName="registry-server" containerID="cri-o://07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c" gracePeriod=2 Jul 24 00:17:35.833917 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:35.833854 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" podStartSLOduration=1.54719356 podStartE2EDuration="2.8338396s" totalImagesPullingTime="1.28664604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:17:35.83033866 +0000 UTC m=+432.562973698" watchObservedRunningTime="2026-07-24 00:17:35.8338396 +0000 UTC m=+432.566474625" Jul 24 00:17:35.845716 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:35.845674 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" podStartSLOduration=1.948372171 podStartE2EDuration="3.845659859s" totalImagesPullingTime="1.897287688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:17:35.843503555 +0000 UTC m=+432.576138580" watchObservedRunningTime="2026-07-24 00:17:35.845659859 +0000 UTC m=+432.578294883" Jul 24 00:17:36.059446 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.059422 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:36.151779 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.151695 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrn6r\" (UniqueName: \"kubernetes.io/projected/b28035e8-9696-43e6-9052-a144e47c0a2d-kube-api-access-vrn6r\") pod \"b28035e8-9696-43e6-9052-a144e47c0a2d\" (UID: \"b28035e8-9696-43e6-9052-a144e47c0a2d\") " Jul 24 00:17:36.153815 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.153792 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28035e8-9696-43e6-9052-a144e47c0a2d-kube-api-access-vrn6r" (OuterVolumeSpecName: "kube-api-access-vrn6r") pod "b28035e8-9696-43e6-9052-a144e47c0a2d" (UID: "b28035e8-9696-43e6-9052-a144e47c0a2d"). InnerVolumeSpecName "kube-api-access-vrn6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:17:36.253224 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.253189 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrn6r\" (UniqueName: \"kubernetes.io/projected/b28035e8-9696-43e6-9052-a144e47c0a2d-kube-api-access-vrn6r\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:36.822720 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.822687 2585 generic.go:356] "Generic (PLEG): container finished" podID="b28035e8-9696-43e6-9052-a144e47c0a2d" containerID="07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c" exitCode=0 Jul 24 00:17:36.823132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.822730 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" Jul 24 00:17:36.823132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.822775 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" event={"ID":"b28035e8-9696-43e6-9052-a144e47c0a2d","Type":"ContainerDied","Data":"07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c"} Jul 24 00:17:36.823132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.822822 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6bkw6" event={"ID":"b28035e8-9696-43e6-9052-a144e47c0a2d","Type":"ContainerDied","Data":"14839047fc08d8a2755bf72ed9dbd49bf62a74a6c3c746b2c649c2c02e8af38b"} Jul 24 00:17:36.823132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.822847 2585 scope.go:117] "RemoveContainer" containerID="07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c" Jul 24 00:17:36.832592 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.832575 2585 scope.go:117] "RemoveContainer" containerID="07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c" Jul 24 00:17:36.832848 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:17:36.832829 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c\": container with ID starting with 07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c not found: ID does not exist" containerID="07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c" Jul 24 00:17:36.832998 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.832968 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c"} err="failed to get container status \"07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c\": rpc error: code = NotFound desc = could not find container \"07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c\": container with ID starting with 07b05bcae75bc5b81c64bca581d197c7aeb26ef27d5560102c76025f04953e3c not found: ID does not exist" Jul 24 00:17:36.844392 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.844369 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bkw6"] Jul 24 00:17:36.849803 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:36.849780 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6bkw6"] Jul 24 00:17:37.798980 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:37.798942 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28035e8-9696-43e6-9052-a144e47c0a2d" path="/var/lib/kubelet/pods/b28035e8-9696-43e6-9052-a144e47c0a2d/volumes" Jul 24 00:17:43.384753 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:43.384703 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:43.384753 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:43.384768 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:43.406215 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:43.406190 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:43.876591 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:43.876565 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-fgfqk" Jul 24 00:17:52.064977 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.064900 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68"] Jul 24 00:17:52.065412 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.065261 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28035e8-9696-43e6-9052-a144e47c0a2d" containerName="registry-server" Jul 24 00:17:52.065412 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.065274 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28035e8-9696-43e6-9052-a144e47c0a2d" containerName="registry-server" Jul 24 00:17:52.065412 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.065339 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28035e8-9696-43e6-9052-a144e47c0a2d" containerName="registry-server" Jul 24 00:17:52.068386 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.068370 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.070930 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.070909 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qm5wl\"" Jul 24 00:17:52.075570 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.075523 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68"] Jul 24 00:17:52.168339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.166501 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c"] Jul 24 00:17:52.171917 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.171893 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.176722 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.176696 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c"] Jul 24 00:17:52.184454 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.184427 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg242\" (UniqueName: \"kubernetes.io/projected/feb330f7-a602-4a9c-8b62-031ac0c65c53-kube-api-access-tg242\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.184579 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.184480 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.184579 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.184502 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.285044 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285012 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.285220 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.285220 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.285220 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285097 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.285220 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lml\" (UniqueName: \"kubernetes.io/projected/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-kube-api-access-c7lml\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.285377 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285222 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg242\" (UniqueName: \"kubernetes.io/projected/feb330f7-a602-4a9c-8b62-031ac0c65c53-kube-api-access-tg242\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.285503 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285482 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.285537 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.285493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.292874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.292827 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg242\" (UniqueName: \"kubernetes.io/projected/feb330f7-a602-4a9c-8b62-031ac0c65c53-kube-api-access-tg242\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.378733 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.378644 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:52.385844 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.385823 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.385964 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.385872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.386031 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.386008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lml\" (UniqueName: \"kubernetes.io/projected/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-kube-api-access-c7lml\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.386226 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.386206 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.386299 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.386264 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.394532 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.394506 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lml\" (UniqueName: \"kubernetes.io/projected/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-kube-api-access-c7lml\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.466398 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.466371 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx"] Jul 24 00:17:52.471594 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.471570 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.478207 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.478158 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx"] Jul 24 00:17:52.481561 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.481537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:52.517372 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.517344 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68"] Jul 24 00:17:52.518990 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:17:52.518966 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb330f7_a602_4a9c_8b62_031ac0c65c53.slice/crio-0f12f2188155ca8e30b77b0feb4daf153d6502366f6d26694a0187133c45b6dd WatchSource:0}: Error finding container 0f12f2188155ca8e30b77b0feb4daf153d6502366f6d26694a0187133c45b6dd: Status 404 returned error can't find the container with id 0f12f2188155ca8e30b77b0feb4daf153d6502366f6d26694a0187133c45b6dd Jul 24 00:17:52.586981 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.586950 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.587134 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.586999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.587199 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.587121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9pqf\" (UniqueName: \"kubernetes.io/projected/98217b70-405b-44f1-98aa-0772e459ff68-kube-api-access-d9pqf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.620544 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.620492 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c"] Jul 24 00:17:52.622659 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:17:52.622628 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27793db7_7125_4f99_b2d6_e6f0a4cdc4ab.slice/crio-b5038fee73fd5cf87ce2395e481c0958b7df7e183a10b12e3cdb1b64a27ebf56 WatchSource:0}: Error finding container b5038fee73fd5cf87ce2395e481c0958b7df7e183a10b12e3cdb1b64a27ebf56: Status 404 returned error can't find the container with id b5038fee73fd5cf87ce2395e481c0958b7df7e183a10b12e3cdb1b64a27ebf56 Jul 24 00:17:52.687856 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.687831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9pqf\" (UniqueName: \"kubernetes.io/projected/98217b70-405b-44f1-98aa-0772e459ff68-kube-api-access-d9pqf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.688025 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.687924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.688025 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.687961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.688334 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.688316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.688387 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.688368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.695511 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.695490 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9pqf\" (UniqueName: \"kubernetes.io/projected/98217b70-405b-44f1-98aa-0772e459ff68-kube-api-access-d9pqf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.785269 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.785234 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:52.866899 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.866851 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt"] Jul 24 00:17:52.871765 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.871738 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:52.877308 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.877281 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt"] Jul 24 00:17:52.902821 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.902733 2585 generic.go:356] "Generic (PLEG): container finished" podID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerID="ed31aab8fdf01baaec472e60cf0bbf693c09c70d36305c5f85b92234ec636dba" exitCode=0 Jul 24 00:17:52.902981 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.902809 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" event={"ID":"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab","Type":"ContainerDied","Data":"ed31aab8fdf01baaec472e60cf0bbf693c09c70d36305c5f85b92234ec636dba"} Jul 24 00:17:52.903046 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.903023 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" event={"ID":"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab","Type":"ContainerStarted","Data":"b5038fee73fd5cf87ce2395e481c0958b7df7e183a10b12e3cdb1b64a27ebf56"} Jul 24 00:17:52.904918 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.904853 2585 generic.go:356] "Generic (PLEG): container finished" podID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerID="de73b1f41a4675a6d14205b535da79098a2ad237a8a20338624eb228752689c9" exitCode=0 Jul 24 00:17:52.904918 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.904896 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" event={"ID":"feb330f7-a602-4a9c-8b62-031ac0c65c53","Type":"ContainerDied","Data":"de73b1f41a4675a6d14205b535da79098a2ad237a8a20338624eb228752689c9"} Jul 24 00:17:52.905097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.904946 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" event={"ID":"feb330f7-a602-4a9c-8b62-031ac0c65c53","Type":"ContainerStarted","Data":"0f12f2188155ca8e30b77b0feb4daf153d6502366f6d26694a0187133c45b6dd"} Jul 24 00:17:52.924678 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.924657 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx"] Jul 24 00:17:52.926769 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:17:52.926744 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98217b70_405b_44f1_98aa_0772e459ff68.slice/crio-9dfe39f7e08b9b53b349033db14afb8a7f8ecf6ac22dd44459a9f22d662f0614 WatchSource:0}: Error finding container 9dfe39f7e08b9b53b349033db14afb8a7f8ecf6ac22dd44459a9f22d662f0614: Status 404 returned error can't find the container with id 9dfe39f7e08b9b53b349033db14afb8a7f8ecf6ac22dd44459a9f22d662f0614 Jul 24 00:17:52.990907 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.990883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8mwn\" (UniqueName: \"kubernetes.io/projected/2088bf82-d4fd-4391-a56f-22fd4539fb81-kube-api-access-h8mwn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:52.991022 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.990928 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:52.991087 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:52.991025 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.092378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.092345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8mwn\" (UniqueName: \"kubernetes.io/projected/2088bf82-d4fd-4391-a56f-22fd4539fb81-kube-api-access-h8mwn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.092739 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.092394 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.092739 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.092425 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.092739 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.092731 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.092847 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.092757 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.100428 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.100409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8mwn\" (UniqueName: \"kubernetes.io/projected/2088bf82-d4fd-4391-a56f-22fd4539fb81-kube-api-access-h8mwn\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.182964 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.182886 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:53.314476 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.314453 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt"] Jul 24 00:17:53.316054 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:17:53.316026 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2088bf82_d4fd_4391_a56f_22fd4539fb81.slice/crio-982506204b941e4c031ef8216ba707a9a3c9366e5bd5ffe0c138fbc19434dbf7 WatchSource:0}: Error finding container 982506204b941e4c031ef8216ba707a9a3c9366e5bd5ffe0c138fbc19434dbf7: Status 404 returned error can't find the container with id 982506204b941e4c031ef8216ba707a9a3c9366e5bd5ffe0c138fbc19434dbf7 Jul 24 00:17:53.911509 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.911477 2585 generic.go:356] "Generic (PLEG): container finished" podID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerID="893a046d3bacb7e98e601eb162861d46c17974c6ad16482c3a3725459f26281c" exitCode=0 Jul 24 00:17:53.911680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.911568 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" event={"ID":"2088bf82-d4fd-4391-a56f-22fd4539fb81","Type":"ContainerDied","Data":"893a046d3bacb7e98e601eb162861d46c17974c6ad16482c3a3725459f26281c"} Jul 24 00:17:53.911680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.911637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" event={"ID":"2088bf82-d4fd-4391-a56f-22fd4539fb81","Type":"ContainerStarted","Data":"982506204b941e4c031ef8216ba707a9a3c9366e5bd5ffe0c138fbc19434dbf7"} Jul 24 00:17:53.913217 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.913193 2585 generic.go:356] "Generic (PLEG): container finished" podID="98217b70-405b-44f1-98aa-0772e459ff68" containerID="96480d1e4cc6f93589830b82707da89c24a50e193f878ae853beb2d0a4c33e41" exitCode=0 Jul 24 00:17:53.913326 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.913268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" event={"ID":"98217b70-405b-44f1-98aa-0772e459ff68","Type":"ContainerDied","Data":"96480d1e4cc6f93589830b82707da89c24a50e193f878ae853beb2d0a4c33e41"} Jul 24 00:17:53.913326 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:53.913304 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" event={"ID":"98217b70-405b-44f1-98aa-0772e459ff68","Type":"ContainerStarted","Data":"9dfe39f7e08b9b53b349033db14afb8a7f8ecf6ac22dd44459a9f22d662f0614"} Jul 24 00:17:54.920442 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.920410 2585 generic.go:356] "Generic (PLEG): container finished" podID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerID="aa725cebcccdef082ef03547de1d96dcf46f11f2034c5ee7f777da0fd4031e73" exitCode=0 Jul 24 00:17:54.920846 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.920473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" event={"ID":"2088bf82-d4fd-4391-a56f-22fd4539fb81","Type":"ContainerDied","Data":"aa725cebcccdef082ef03547de1d96dcf46f11f2034c5ee7f777da0fd4031e73"} Jul 24 00:17:54.922369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.922337 2585 generic.go:356] "Generic (PLEG): container finished" podID="98217b70-405b-44f1-98aa-0772e459ff68" containerID="97976102d60b6cdfe2c594938680e25825ba2bac0ea532e91f797f406cc9ba03" exitCode=0 Jul 24 00:17:54.922446 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.922368 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" event={"ID":"98217b70-405b-44f1-98aa-0772e459ff68","Type":"ContainerDied","Data":"97976102d60b6cdfe2c594938680e25825ba2bac0ea532e91f797f406cc9ba03"} Jul 24 00:17:54.924391 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.924369 2585 generic.go:356] "Generic (PLEG): container finished" podID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerID="e92ed4e58d7d5e4d5ec6b7d8eba285f7505b30f09f08cf4db03dc6d54e04ee40" exitCode=0 Jul 24 00:17:54.924478 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.924436 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" event={"ID":"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab","Type":"ContainerDied","Data":"e92ed4e58d7d5e4d5ec6b7d8eba285f7505b30f09f08cf4db03dc6d54e04ee40"} Jul 24 00:17:54.926299 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.926276 2585 generic.go:356] "Generic (PLEG): container finished" podID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerID="02282e58c50b95beb359718f0679eb4101a8c20a5ad47c09cd9044535c972c75" exitCode=0 Jul 24 00:17:54.926370 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:54.926314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" event={"ID":"feb330f7-a602-4a9c-8b62-031ac0c65c53","Type":"ContainerDied","Data":"02282e58c50b95beb359718f0679eb4101a8c20a5ad47c09cd9044535c972c75"} Jul 24 00:17:55.933746 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.933707 2585 generic.go:356] "Generic (PLEG): container finished" podID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerID="0d4dba1d961c9e13fd63e35647c843b50c8c003b5f1fca0dc75326cfdff16c6b" exitCode=0 Jul 24 00:17:55.934205 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.933766 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" event={"ID":"2088bf82-d4fd-4391-a56f-22fd4539fb81","Type":"ContainerDied","Data":"0d4dba1d961c9e13fd63e35647c843b50c8c003b5f1fca0dc75326cfdff16c6b"} Jul 24 00:17:55.935721 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.935698 2585 generic.go:356] "Generic (PLEG): container finished" podID="98217b70-405b-44f1-98aa-0772e459ff68" containerID="568befd9dc311d3cff9b4a14f2c9eaecac3d6122cc6b4a920ebe5d1adcf8b733" exitCode=0 Jul 24 00:17:55.935851 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.935770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" event={"ID":"98217b70-405b-44f1-98aa-0772e459ff68","Type":"ContainerDied","Data":"568befd9dc311d3cff9b4a14f2c9eaecac3d6122cc6b4a920ebe5d1adcf8b733"} Jul 24 00:17:55.937728 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.937700 2585 generic.go:356] "Generic (PLEG): container finished" podID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerID="8f0a1c0b212ee8f8d38e6b27b651f77b157710fbb35aa0ddff4f2e56216f29f3" exitCode=0 Jul 24 00:17:55.937833 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.937770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" event={"ID":"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab","Type":"ContainerDied","Data":"8f0a1c0b212ee8f8d38e6b27b651f77b157710fbb35aa0ddff4f2e56216f29f3"} Jul 24 00:17:55.939690 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.939670 2585 generic.go:356] "Generic (PLEG): container finished" podID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerID="c14769c2db26d96727ee462189f382e95e58522323b01b97421df40e2075e358" exitCode=0 Jul 24 00:17:55.939789 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:55.939708 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" event={"ID":"feb330f7-a602-4a9c-8b62-031ac0c65c53","Type":"ContainerDied","Data":"c14769c2db26d96727ee462189f382e95e58522323b01b97421df40e2075e358"} Jul 24 00:17:57.076829 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.076800 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:57.128888 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.128842 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-util\") pod \"2088bf82-d4fd-4391-a56f-22fd4539fb81\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " Jul 24 00:17:57.129039 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.129019 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8mwn\" (UniqueName: \"kubernetes.io/projected/2088bf82-d4fd-4391-a56f-22fd4539fb81-kube-api-access-h8mwn\") pod \"2088bf82-d4fd-4391-a56f-22fd4539fb81\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " Jul 24 00:17:57.129132 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.129053 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-bundle\") pod \"2088bf82-d4fd-4391-a56f-22fd4539fb81\" (UID: \"2088bf82-d4fd-4391-a56f-22fd4539fb81\") " Jul 24 00:17:57.129674 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.129651 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-bundle" (OuterVolumeSpecName: "bundle") pod "2088bf82-d4fd-4391-a56f-22fd4539fb81" (UID: "2088bf82-d4fd-4391-a56f-22fd4539fb81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.132452 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.132428 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2088bf82-d4fd-4391-a56f-22fd4539fb81-kube-api-access-h8mwn" (OuterVolumeSpecName: "kube-api-access-h8mwn") pod "2088bf82-d4fd-4391-a56f-22fd4539fb81" (UID: "2088bf82-d4fd-4391-a56f-22fd4539fb81"). InnerVolumeSpecName "kube-api-access-h8mwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:17:57.132965 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.132951 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:57.134881 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.134847 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-util" (OuterVolumeSpecName: "util") pod "2088bf82-d4fd-4391-a56f-22fd4539fb81" (UID: "2088bf82-d4fd-4391-a56f-22fd4539fb81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.142308 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.142291 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:57.150503 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.150480 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:57.229542 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229456 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-bundle\") pod \"feb330f7-a602-4a9c-8b62-031ac0c65c53\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " Jul 24 00:17:57.229542 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229504 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg242\" (UniqueName: \"kubernetes.io/projected/feb330f7-a602-4a9c-8b62-031ac0c65c53-kube-api-access-tg242\") pod \"feb330f7-a602-4a9c-8b62-031ac0c65c53\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " Jul 24 00:17:57.229764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229557 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-bundle\") pod \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " Jul 24 00:17:57.229764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229601 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-bundle\") pod \"98217b70-405b-44f1-98aa-0772e459ff68\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " Jul 24 00:17:57.229764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229626 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9pqf\" (UniqueName: \"kubernetes.io/projected/98217b70-405b-44f1-98aa-0772e459ff68-kube-api-access-d9pqf\") pod \"98217b70-405b-44f1-98aa-0772e459ff68\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " Jul 24 00:17:57.229764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229681 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-util\") pod \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " Jul 24 00:17:57.229764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229712 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-util\") pod \"feb330f7-a602-4a9c-8b62-031ac0c65c53\" (UID: \"feb330f7-a602-4a9c-8b62-031ac0c65c53\") " Jul 24 00:17:57.229764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229736 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-util\") pod \"98217b70-405b-44f1-98aa-0772e459ff68\" (UID: \"98217b70-405b-44f1-98aa-0772e459ff68\") " Jul 24 00:17:57.230145 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.229773 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7lml\" (UniqueName: \"kubernetes.io/projected/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-kube-api-access-c7lml\") pod \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\" (UID: \"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab\") " Jul 24 00:17:57.230145 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.230033 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8mwn\" (UniqueName: \"kubernetes.io/projected/2088bf82-d4fd-4391-a56f-22fd4539fb81-kube-api-access-h8mwn\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.230145 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.230054 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.230145 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.230068 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2088bf82-d4fd-4391-a56f-22fd4539fb81-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.230453 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.230419 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-bundle" (OuterVolumeSpecName: "bundle") pod "98217b70-405b-44f1-98aa-0772e459ff68" (UID: "98217b70-405b-44f1-98aa-0772e459ff68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.230543 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.230475 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-bundle" (OuterVolumeSpecName: "bundle") pod "27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" (UID: "27793db7-7125-4f99-b2d6-e6f0a4cdc4ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.230799 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.230773 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-bundle" (OuterVolumeSpecName: "bundle") pod "feb330f7-a602-4a9c-8b62-031ac0c65c53" (UID: "feb330f7-a602-4a9c-8b62-031ac0c65c53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.232133 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.232094 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98217b70-405b-44f1-98aa-0772e459ff68-kube-api-access-d9pqf" (OuterVolumeSpecName: "kube-api-access-d9pqf") pod "98217b70-405b-44f1-98aa-0772e459ff68" (UID: "98217b70-405b-44f1-98aa-0772e459ff68"). InnerVolumeSpecName "kube-api-access-d9pqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:17:57.232222 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.232165 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb330f7-a602-4a9c-8b62-031ac0c65c53-kube-api-access-tg242" (OuterVolumeSpecName: "kube-api-access-tg242") pod "feb330f7-a602-4a9c-8b62-031ac0c65c53" (UID: "feb330f7-a602-4a9c-8b62-031ac0c65c53"). InnerVolumeSpecName "kube-api-access-tg242". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:17:57.232222 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.232162 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-kube-api-access-c7lml" (OuterVolumeSpecName: "kube-api-access-c7lml") pod "27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" (UID: "27793db7-7125-4f99-b2d6-e6f0a4cdc4ab"). InnerVolumeSpecName "kube-api-access-c7lml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:17:57.237247 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.237223 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-util" (OuterVolumeSpecName: "util") pod "feb330f7-a602-4a9c-8b62-031ac0c65c53" (UID: "feb330f7-a602-4a9c-8b62-031ac0c65c53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.237609 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.237582 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-util" (OuterVolumeSpecName: "util") pod "27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" (UID: "27793db7-7125-4f99-b2d6-e6f0a4cdc4ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.238846 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.238829 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-util" (OuterVolumeSpecName: "util") pod "98217b70-405b-44f1-98aa-0772e459ff68" (UID: "98217b70-405b-44f1-98aa-0772e459ff68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:17:57.330457 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330421 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg242\" (UniqueName: \"kubernetes.io/projected/feb330f7-a602-4a9c-8b62-031ac0c65c53-kube-api-access-tg242\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330457 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330451 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330457 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330462 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330471 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9pqf\" (UniqueName: \"kubernetes.io/projected/98217b70-405b-44f1-98aa-0772e459ff68-kube-api-access-d9pqf\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330480 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330489 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330496 2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/98217b70-405b-44f1-98aa-0772e459ff68-util\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330503 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7lml\" (UniqueName: \"kubernetes.io/projected/27793db7-7125-4f99-b2d6-e6f0a4cdc4ab-kube-api-access-c7lml\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.330683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.330510 2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feb330f7-a602-4a9c-8b62-031ac0c65c53-bundle\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:17:57.951808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.951778 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" Jul 24 00:17:57.952007 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.951777 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt" event={"ID":"2088bf82-d4fd-4391-a56f-22fd4539fb81","Type":"ContainerDied","Data":"982506204b941e4c031ef8216ba707a9a3c9366e5bd5ffe0c138fbc19434dbf7"} Jul 24 00:17:57.952007 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.951914 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982506204b941e4c031ef8216ba707a9a3c9366e5bd5ffe0c138fbc19434dbf7" Jul 24 00:17:57.953925 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.953898 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" event={"ID":"98217b70-405b-44f1-98aa-0772e459ff68","Type":"ContainerDied","Data":"9dfe39f7e08b9b53b349033db14afb8a7f8ecf6ac22dd44459a9f22d662f0614"} Jul 24 00:17:57.954066 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.953941 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dfe39f7e08b9b53b349033db14afb8a7f8ecf6ac22dd44459a9f22d662f0614" Jul 24 00:17:57.954066 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.953995 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx" Jul 24 00:17:57.956057 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.956027 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" Jul 24 00:17:57.956172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.956027 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c" event={"ID":"27793db7-7125-4f99-b2d6-e6f0a4cdc4ab","Type":"ContainerDied","Data":"b5038fee73fd5cf87ce2395e481c0958b7df7e183a10b12e3cdb1b64a27ebf56"} Jul 24 00:17:57.956172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.956165 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5038fee73fd5cf87ce2395e481c0958b7df7e183a10b12e3cdb1b64a27ebf56" Jul 24 00:17:57.958094 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.958073 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" Jul 24 00:17:57.958094 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.958083 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68" event={"ID":"feb330f7-a602-4a9c-8b62-031ac0c65c53","Type":"ContainerDied","Data":"0f12f2188155ca8e30b77b0feb4daf153d6502366f6d26694a0187133c45b6dd"} Jul 24 00:17:57.958235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:17:57.958119 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f12f2188155ca8e30b77b0feb4daf153d6502366f6d26694a0187133c45b6dd" Jul 24 00:18:02.903022 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.902989 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct"] Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903329 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="extract" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903342 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="extract" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903356 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="pull" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903362 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="pull" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903368 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="extract" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903373 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="extract" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903380 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="extract" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903386 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="extract" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903395 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="pull" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903400 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="pull" Jul 24 00:18:02.903401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903408 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903413 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903421 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903429 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903437 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="pull" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903443 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="pull" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903447 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="pull" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903452 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="pull" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903458 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="extract" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903462 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="extract" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903472 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903477 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903488 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903492 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="util" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903554 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="feb330f7-a602-4a9c-8b62-031ac0c65c53" containerName="extract" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903562 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="98217b70-405b-44f1-98aa-0772e459ff68" containerName="extract" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903567 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="27793db7-7125-4f99-b2d6-e6f0a4cdc4ab" containerName="extract" Jul 24 00:18:02.903772 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.903574 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2088bf82-d4fd-4391-a56f-22fd4539fb81" containerName="extract" Jul 24 00:18:02.907874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.907848 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:02.910324 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.910304 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 00:18:02.910436 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.910356 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-22kpm\"" Jul 24 00:18:02.918646 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.918621 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct"] Jul 24 00:18:02.980818 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:02.980783 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhhz\" (UniqueName: \"kubernetes.io/projected/210accd6-b966-48bf-b346-f35109aaa8f9-kube-api-access-sxhhz\") pod \"dns-operator-controller-manager-864dd78c4c-th6ct\" (UID: \"210accd6-b966-48bf-b346-f35109aaa8f9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:03.082161 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.082127 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhhz\" (UniqueName: \"kubernetes.io/projected/210accd6-b966-48bf-b346-f35109aaa8f9-kube-api-access-sxhhz\") pod \"dns-operator-controller-manager-864dd78c4c-th6ct\" (UID: \"210accd6-b966-48bf-b346-f35109aaa8f9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:03.092345 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.092316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhhz\" (UniqueName: \"kubernetes.io/projected/210accd6-b966-48bf-b346-f35109aaa8f9-kube-api-access-sxhhz\") pod \"dns-operator-controller-manager-864dd78c4c-th6ct\" (UID: \"210accd6-b966-48bf-b346-f35109aaa8f9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:03.218453 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.218348 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:03.348762 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.348736 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct"] Jul 24 00:18:03.351384 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:03.351358 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210accd6_b966_48bf_b346_f35109aaa8f9.slice/crio-a17f56b1fe29437c16d46af82dfd0e8748a32aedf0d488337be25ecc0e7d8a86 WatchSource:0}: Error finding container a17f56b1fe29437c16d46af82dfd0e8748a32aedf0d488337be25ecc0e7d8a86: Status 404 returned error can't find the container with id a17f56b1fe29437c16d46af82dfd0e8748a32aedf0d488337be25ecc0e7d8a86 Jul 24 00:18:03.954172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.954131 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-674849cc4f-jlpnv"] Jul 24 00:18:03.958439 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.958417 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:03.961323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.961245 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 00:18:03.961323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.961253 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 00:18:03.961491 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.961253 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 00:18:03.962318 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.962301 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7nc75\"" Jul 24 00:18:03.962642 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.962626 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 00:18:03.962757 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.962647 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 00:18:03.962757 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.962673 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 00:18:03.962757 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.962731 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 00:18:03.967393 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.967292 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 00:18:03.970742 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.970721 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674849cc4f-jlpnv"] Jul 24 00:18:03.993099 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:03.993064 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" event={"ID":"210accd6-b966-48bf-b346-f35109aaa8f9","Type":"ContainerStarted","Data":"a17f56b1fe29437c16d46af82dfd0e8748a32aedf0d488337be25ecc0e7d8a86"} Jul 24 00:18:04.089956 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.089918 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-console-config\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.090103 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.089986 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-trusted-ca-bundle\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.090103 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.090019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32aa6363-aae9-4100-915d-751b74d4beb3-console-oauth-config\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.090103 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.090044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hftmh\" (UniqueName: \"kubernetes.io/projected/32aa6363-aae9-4100-915d-751b74d4beb3-kube-api-access-hftmh\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.090103 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.090101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-service-ca\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.090285 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.090135 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-oauth-serving-cert\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.090285 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.090183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32aa6363-aae9-4100-915d-751b74d4beb3-console-serving-cert\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191345 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191306 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32aa6363-aae9-4100-915d-751b74d4beb3-console-serving-cert\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191525 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191481 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-console-config\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191591 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191551 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-trusted-ca-bundle\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191643 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32aa6363-aae9-4100-915d-751b74d4beb3-console-oauth-config\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191643 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hftmh\" (UniqueName: \"kubernetes.io/projected/32aa6363-aae9-4100-915d-751b74d4beb3-kube-api-access-hftmh\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191723 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-service-ca\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.191723 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.191698 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-oauth-serving-cert\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.192394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.192244 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-console-config\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.192506 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.192441 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-oauth-serving-cert\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.192578 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.192521 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-service-ca\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.192727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.192699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aa6363-aae9-4100-915d-751b74d4beb3-trusted-ca-bundle\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.194232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.194202 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32aa6363-aae9-4100-915d-751b74d4beb3-console-serving-cert\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.194630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.194608 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32aa6363-aae9-4100-915d-751b74d4beb3-console-oauth-config\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.206169 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.206105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hftmh\" (UniqueName: \"kubernetes.io/projected/32aa6363-aae9-4100-915d-751b74d4beb3-kube-api-access-hftmh\") pod \"console-674849cc4f-jlpnv\" (UID: \"32aa6363-aae9-4100-915d-751b74d4beb3\") " pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.270812 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.270776 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:04.707959 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:04.707930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674849cc4f-jlpnv"] Jul 24 00:18:04.708718 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:04.708690 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32aa6363_aae9_4100_915d_751b74d4beb3.slice/crio-bec425a4275d53fb7f91a0d36909279a8e287876b499c8a014b7f562e1147a37 WatchSource:0}: Error finding container bec425a4275d53fb7f91a0d36909279a8e287876b499c8a014b7f562e1147a37: Status 404 returned error can't find the container with id bec425a4275d53fb7f91a0d36909279a8e287876b499c8a014b7f562e1147a37 Jul 24 00:18:05.001250 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:05.001167 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674849cc4f-jlpnv" event={"ID":"32aa6363-aae9-4100-915d-751b74d4beb3","Type":"ContainerStarted","Data":"bec425a4275d53fb7f91a0d36909279a8e287876b499c8a014b7f562e1147a37"} Jul 24 00:18:06.009947 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.009905 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" event={"ID":"210accd6-b966-48bf-b346-f35109aaa8f9","Type":"ContainerStarted","Data":"292d8893bbce4f7c1a0c44ed26a457f916b0522dc85799723d8263e636d92bcd"} Jul 24 00:18:06.010458 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.010018 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:06.026583 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.026536 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" podStartSLOduration=2.102442068 podStartE2EDuration="4.026521424s" totalImagesPullingTime="1.924079356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:06.025463065 +0000 UTC m=+462.758098121" watchObservedRunningTime="2026-07-24 00:18:06.026521424 +0000 UTC m=+462.759156449" Jul 24 00:18:06.368311 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.368272 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj"] Jul 24 00:18:06.372252 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.372225 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:06.374917 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.374892 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hbkhv\"" Jul 24 00:18:06.382817 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.382796 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj"] Jul 24 00:18:06.513839 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.513804 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4t89\" (UniqueName: \"kubernetes.io/projected/d8013523-aa05-4626-80c8-0d178a38d456-kube-api-access-v4t89\") pod \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" (UID: \"d8013523-aa05-4626-80c8-0d178a38d456\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:06.614578 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.614541 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4t89\" (UniqueName: \"kubernetes.io/projected/d8013523-aa05-4626-80c8-0d178a38d456-kube-api-access-v4t89\") pod \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" (UID: \"d8013523-aa05-4626-80c8-0d178a38d456\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:06.625339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.625271 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4t89\" (UniqueName: \"kubernetes.io/projected/d8013523-aa05-4626-80c8-0d178a38d456-kube-api-access-v4t89\") pod \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" (UID: \"d8013523-aa05-4626-80c8-0d178a38d456\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:06.687302 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.687263 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:06.860776 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:06.860736 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj"] Jul 24 00:18:06.864003 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:06.863975 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8013523_aa05_4626_80c8_0d178a38d456.slice/crio-3cb771a39fc722a95fe06d5c34e2f500c284389415dd222f5bddc903318c201a WatchSource:0}: Error finding container 3cb771a39fc722a95fe06d5c34e2f500c284389415dd222f5bddc903318c201a: Status 404 returned error can't find the container with id 3cb771a39fc722a95fe06d5c34e2f500c284389415dd222f5bddc903318c201a Jul 24 00:18:07.017541 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:07.017456 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" event={"ID":"d8013523-aa05-4626-80c8-0d178a38d456","Type":"ContainerStarted","Data":"3cb771a39fc722a95fe06d5c34e2f500c284389415dd222f5bddc903318c201a"} Jul 24 00:18:09.032339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:09.032304 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674849cc4f-jlpnv" event={"ID":"32aa6363-aae9-4100-915d-751b74d4beb3","Type":"ContainerStarted","Data":"8277465ce12f888665dd0740cbea4be327ed72ccba8edbc1a589e9c511078661"} Jul 24 00:18:09.051577 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:09.051520 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-674849cc4f-jlpnv" podStartSLOduration=2.529596622 podStartE2EDuration="6.051499512s" totalImagesPullingTime="3.52190289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:09.049547842 +0000 UTC m=+465.782182911" watchObservedRunningTime="2026-07-24 00:18:09.051499512 +0000 UTC m=+465.784134537" Jul 24 00:18:10.039439 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:10.039400 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" event={"ID":"d8013523-aa05-4626-80c8-0d178a38d456","Type":"ContainerStarted","Data":"8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a"} Jul 24 00:18:10.039829 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:10.039709 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:10.058975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:10.058921 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" podStartSLOduration=1.657739566 podStartE2EDuration="4.058906899s" totalImagesPullingTime="2.401167333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:10.055769651 +0000 UTC m=+466.788404699" watchObservedRunningTime="2026-07-24 00:18:10.058906899 +0000 UTC m=+466.791541989" Jul 24 00:18:11.176555 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.176521 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw"] Jul 24 00:18:11.180017 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.180000 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.182641 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.182620 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-48nbf\"" Jul 24 00:18:11.190518 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.190495 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw"] Jul 24 00:18:11.363761 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.363708 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbb5f\" (UniqueName: \"kubernetes.io/projected/21e0541a-acad-4d27-aef8-9ec20350f9b5-kube-api-access-gbb5f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.363989 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.363843 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21e0541a-acad-4d27-aef8-9ec20350f9b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.464658 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.464571 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21e0541a-acad-4d27-aef8-9ec20350f9b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.464658 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.464632 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbb5f\" (UniqueName: \"kubernetes.io/projected/21e0541a-acad-4d27-aef8-9ec20350f9b5-kube-api-access-gbb5f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.464986 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.464968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21e0541a-acad-4d27-aef8-9ec20350f9b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.475949 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.475924 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbb5f\" (UniqueName: \"kubernetes.io/projected/21e0541a-acad-4d27-aef8-9ec20350f9b5-kube-api-access-gbb5f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.490904 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.490855 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:11.648773 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:11.648749 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw"] Jul 24 00:18:11.650494 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:11.650464 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e0541a_acad_4d27_aef8_9ec20350f9b5.slice/crio-6e0d14e44c4383808341a26ae08f627e689ee9ee943623d5518dd05e42bc59cd WatchSource:0}: Error finding container 6e0d14e44c4383808341a26ae08f627e689ee9ee943623d5518dd05e42bc59cd: Status 404 returned error can't find the container with id 6e0d14e44c4383808341a26ae08f627e689ee9ee943623d5518dd05e42bc59cd Jul 24 00:18:12.052895 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:12.052843 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" event={"ID":"21e0541a-acad-4d27-aef8-9ec20350f9b5","Type":"ContainerStarted","Data":"6e0d14e44c4383808341a26ae08f627e689ee9ee943623d5518dd05e42bc59cd"} Jul 24 00:18:14.271949 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.271911 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:14.272393 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.271985 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:14.278329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.278305 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:14.513568 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.510774 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-656c6"] Jul 24 00:18:14.517164 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.517139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:14.519712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.519679 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q5h29\"" Jul 24 00:18:14.522476 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.522399 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-656c6"] Jul 24 00:18:14.696070 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.696028 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zsp\" (UniqueName: \"kubernetes.io/projected/1324ba49-6b60-407e-85f9-c1ed0d01c917-kube-api-access-66zsp\") pod \"authorino-operator-75cc8d5cb7-656c6\" (UID: \"1324ba49-6b60-407e-85f9-c1ed0d01c917\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:14.796933 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.796829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zsp\" (UniqueName: \"kubernetes.io/projected/1324ba49-6b60-407e-85f9-c1ed0d01c917-kube-api-access-66zsp\") pod \"authorino-operator-75cc8d5cb7-656c6\" (UID: \"1324ba49-6b60-407e-85f9-c1ed0d01c917\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:14.808581 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.808541 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zsp\" (UniqueName: \"kubernetes.io/projected/1324ba49-6b60-407e-85f9-c1ed0d01c917-kube-api-access-66zsp\") pod \"authorino-operator-75cc8d5cb7-656c6\" (UID: \"1324ba49-6b60-407e-85f9-c1ed0d01c917\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:14.829166 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.829134 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:14.978847 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:14.978822 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-656c6"] Jul 24 00:18:14.980800 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:14.980769 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1324ba49_6b60_407e_85f9_c1ed0d01c917.slice/crio-502bd06c929304464871c9bdec90c19ef936a60f1745711b285bbcd721251d3a WatchSource:0}: Error finding container 502bd06c929304464871c9bdec90c19ef936a60f1745711b285bbcd721251d3a: Status 404 returned error can't find the container with id 502bd06c929304464871c9bdec90c19ef936a60f1745711b285bbcd721251d3a Jul 24 00:18:15.072325 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:15.072293 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" event={"ID":"1324ba49-6b60-407e-85f9-c1ed0d01c917","Type":"ContainerStarted","Data":"502bd06c929304464871c9bdec90c19ef936a60f1745711b285bbcd721251d3a"} Jul 24 00:18:15.076419 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:15.076397 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-674849cc4f-jlpnv" Jul 24 00:18:17.019822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:17.019790 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-th6ct" Jul 24 00:18:19.100621 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:19.100586 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" event={"ID":"21e0541a-acad-4d27-aef8-9ec20350f9b5","Type":"ContainerStarted","Data":"85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e"} Jul 24 00:18:19.101101 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:19.100722 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:19.102131 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:19.102107 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" event={"ID":"1324ba49-6b60-407e-85f9-c1ed0d01c917","Type":"ContainerStarted","Data":"7d4f496ecbe7caee321b12244a4d8901ca2b5951f26c1854c659c141f404916f"} Jul 24 00:18:19.102249 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:19.102155 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:19.119754 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:19.119704 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" podStartSLOduration=1.282914394 podStartE2EDuration="8.119685733s" totalImagesPullingTime="6.836771339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:19.11920262 +0000 UTC m=+475.851837648" watchObservedRunningTime="2026-07-24 00:18:19.119685733 +0000 UTC m=+475.852320761" Jul 24 00:18:19.141087 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:19.141040 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" podStartSLOduration=1.646278604 podStartE2EDuration="5.141027199s" totalImagesPullingTime="3.494748595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:19.13872428 +0000 UTC m=+475.871359309" watchObservedRunningTime="2026-07-24 00:18:19.141027199 +0000 UTC m=+475.873662289" Jul 24 00:18:21.047103 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.047073 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:21.270383 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.270352 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk"] Jul 24 00:18:21.274042 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.274023 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.276533 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.276511 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 00:18:21.276653 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.276563 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qm5wl\"" Jul 24 00:18:21.276653 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.276621 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 00:18:21.283377 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.283357 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk"] Jul 24 00:18:21.351600 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.351570 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.351748 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.351610 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.351748 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.351688 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqzs\" (UniqueName: \"kubernetes.io/projected/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-kube-api-access-7dqzs\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.453039 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.453003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqzs\" (UniqueName: \"kubernetes.io/projected/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-kube-api-access-7dqzs\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.453232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.453058 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.453232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.453086 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.453232 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:21.453166 2585 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 00:18:21.453232 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:21.453222 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-plugin-serving-cert podName:3d64bb08-9653-494c-8f76-f1f9c9fee2c7 nodeName:}" failed. No retries permitted until 2026-07-24 00:18:21.953205858 +0000 UTC m=+478.685840862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-z59hk" (UID: "3d64bb08-9653-494c-8f76-f1f9c9fee2c7") : secret "plugin-serving-cert" not found Jul 24 00:18:21.453656 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.453635 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.464564 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.464529 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqzs\" (UniqueName: \"kubernetes.io/projected/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-kube-api-access-7dqzs\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.958628 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.958569 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:21.961201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:21.961177 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d64bb08-9653-494c-8f76-f1f9c9fee2c7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-z59hk\" (UID: \"3d64bb08-9653-494c-8f76-f1f9c9fee2c7\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:22.183806 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:22.183753 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" Jul 24 00:18:22.319437 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:22.319411 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk"] Jul 24 00:18:22.321107 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:22.321084 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d64bb08_9653_494c_8f76_f1f9c9fee2c7.slice/crio-551e9dea1cb25e27b0823a38b426e5e54e8c501a11a30829c078d9bb32b70d98 WatchSource:0}: Error finding container 551e9dea1cb25e27b0823a38b426e5e54e8c501a11a30829c078d9bb32b70d98: Status 404 returned error can't find the container with id 551e9dea1cb25e27b0823a38b426e5e54e8c501a11a30829c078d9bb32b70d98 Jul 24 00:18:23.127520 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:23.127477 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" event={"ID":"3d64bb08-9653-494c-8f76-f1f9c9fee2c7","Type":"ContainerStarted","Data":"551e9dea1cb25e27b0823a38b426e5e54e8c501a11a30829c078d9bb32b70d98"} Jul 24 00:18:30.110575 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:30.110530 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:30.111080 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:30.110806 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-656c6" Jul 24 00:18:31.071194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.071157 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg"] Jul 24 00:18:31.078390 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.078365 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.089294 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.089272 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg"] Jul 24 00:18:31.147238 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.147191 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c70127b-108f-44f4-bc77-0bf30b56dfa9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.147656 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.147258 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9cd\" (UniqueName: \"kubernetes.io/projected/3c70127b-108f-44f4-bc77-0bf30b56dfa9-kube-api-access-5j9cd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.248455 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.248420 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c70127b-108f-44f4-bc77-0bf30b56dfa9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.248636 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.248462 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9cd\" (UniqueName: \"kubernetes.io/projected/3c70127b-108f-44f4-bc77-0bf30b56dfa9-kube-api-access-5j9cd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.248956 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.248928 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c70127b-108f-44f4-bc77-0bf30b56dfa9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.258606 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.258583 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9cd\" (UniqueName: \"kubernetes.io/projected/3c70127b-108f-44f4-bc77-0bf30b56dfa9-kube-api-access-5j9cd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.391642 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.391562 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:31.868573 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.868534 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg"] Jul 24 00:18:31.876172 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.876145 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg"] Jul 24 00:18:31.886935 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.886899 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw"] Jul 24 00:18:31.887312 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.887124 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" containerName="manager" containerID="cri-o://85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e" gracePeriod=2 Jul 24 00:18:31.891635 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.891340 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5"] Jul 24 00:18:31.898624 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.898604 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:31.899959 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.899929 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw"] Jul 24 00:18:31.906583 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:31.906544 2585 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"21e0541a-acad-4d27-aef8-9ec20350f9b5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-42ghw\": pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:31.910409 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:31.910240 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:31.914018 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.913993 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5"] Jul 24 00:18:31.917301 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.917282 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc"] Jul 24 00:18:31.918594 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.918571 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" containerName="manager" Jul 24 00:18:31.918594 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.918596 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" containerName="manager" Jul 24 00:18:31.918747 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.918718 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" containerName="manager" Jul 24 00:18:31.923918 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.923893 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj"] Jul 24 00:18:31.924135 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.924112 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:31.924135 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.924124 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" podUID="d8013523-aa05-4626-80c8-0d178a38d456" containerName="manager" containerID="cri-o://8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a" gracePeriod=2 Jul 24 00:18:31.930353 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.930011 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj"] Jul 24 00:18:31.933066 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.933035 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc"] Jul 24 00:18:31.955943 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.955915 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxd2\" (UniqueName: \"kubernetes.io/projected/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-kube-api-access-lbxd2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rb2m5\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:31.956105 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:31.956061 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rb2m5\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:31.964363 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:31.964316 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:31.966604 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:31.966570 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:32.056925 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.056894 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfvpc\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:32.057082 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.057003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxd2\" (UniqueName: \"kubernetes.io/projected/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-kube-api-access-lbxd2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rb2m5\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:32.057151 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.057118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rb2m5\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:32.057205 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.057149 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wkl\" (UniqueName: \"kubernetes.io/projected/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-kube-api-access-67wkl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfvpc\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:32.057577 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.057538 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rb2m5\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:32.072706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.072678 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxd2\" (UniqueName: \"kubernetes.io/projected/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-kube-api-access-lbxd2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rb2m5\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:32.158450 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.158361 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wkl\" (UniqueName: \"kubernetes.io/projected/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-kube-api-access-67wkl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfvpc\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:32.158450 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.158418 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfvpc\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:32.158986 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.158815 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfvpc\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:32.171032 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.171005 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wkl\" (UniqueName: \"kubernetes.io/projected/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-kube-api-access-67wkl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nfvpc\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:32.329422 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.329382 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:32.335879 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:32.335764 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:33.799032 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:33.798990 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:33.801075 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:33.801039 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.245337 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.244367 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5"] Jul 24 00:18:45.248055 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:45.248025 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e71c577_9dd8_4ead_8d07_6d3fabe9e148.slice/crio-9aff38024162d671b9b0ea3addc4fa9e910ba729ba304cab51f1f91791cf9fcf WatchSource:0}: Error finding container 9aff38024162d671b9b0ea3addc4fa9e910ba729ba304cab51f1f91791cf9fcf: Status 404 returned error can't find the container with id 9aff38024162d671b9b0ea3addc4fa9e910ba729ba304cab51f1f91791cf9fcf Jul 24 00:18:45.251489 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.251463 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.253855 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.253816 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.255809 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.255788 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.257880 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.257835 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.259817 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.259791 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.271518 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.271498 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc"] Jul 24 00:18:45.274921 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.274897 2585 generic.go:356] "Generic (PLEG): container finished" podID="d8013523-aa05-4626-80c8-0d178a38d456" containerID="8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a" exitCode=0 Jul 24 00:18:45.275009 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.274939 2585 scope.go:117] "RemoveContainer" containerID="8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a" Jul 24 00:18:45.275067 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.274940 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.275121 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:45.275070 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac84f105_1bad_4085_9b0f_e0c2bed9b20e.slice/crio-4b3ee4dbb7b244b769c1f532dcdc811f1869bcb3a24e8a6746ae1e7c68eca725 WatchSource:0}: Error finding container 4b3ee4dbb7b244b769c1f532dcdc811f1869bcb3a24e8a6746ae1e7c68eca725: Status 404 returned error can't find the container with id 4b3ee4dbb7b244b769c1f532dcdc811f1869bcb3a24e8a6746ae1e7c68eca725 Jul 24 00:18:45.276510 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.276487 2585 generic.go:356] "Generic (PLEG): container finished" podID="21e0541a-acad-4d27-aef8-9ec20350f9b5" containerID="85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e" exitCode=0 Jul 24 00:18:45.276607 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.276525 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.277151 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.277125 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.278334 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.278306 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" event={"ID":"3d64bb08-9653-494c-8f76-f1f9c9fee2c7","Type":"ContainerStarted","Data":"57c356a3331a7aca0b80974b5de690f34d2d01035b30fca285ec19fa9f24e706"} Jul 24 00:18:45.278996 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.278970 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.279633 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.279615 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" event={"ID":"5e71c577-9dd8-4ead-8d07-6d3fabe9e148","Type":"ContainerStarted","Data":"9aff38024162d671b9b0ea3addc4fa9e910ba729ba304cab51f1f91791cf9fcf"} Jul 24 00:18:45.281016 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.280878 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.286181 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.286162 2585 scope.go:117] "RemoveContainer" containerID="8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a" Jul 24 00:18:45.286472 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.286454 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a\": container with ID starting with 8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a not found: ID does not exist" containerID="8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a" Jul 24 00:18:45.286518 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.286481 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a"} err="failed to get container status \"8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a\": rpc error: code = NotFound desc = could not find container \"8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a\": container with ID starting with 8101703846f9070580ce3ab1c989374df1872a95354d046d9f438d0b177e254a not found: ID does not exist" Jul 24 00:18:45.286518 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.286498 2585 scope.go:117] "RemoveContainer" containerID="85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e" Jul 24 00:18:45.296432 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.296402 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.296741 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.296724 2585 scope.go:117] "RemoveContainer" containerID="85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e" Jul 24 00:18:45.296811 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.296706 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-z59hk" podStartSLOduration=1.445236687 podStartE2EDuration="24.296690604s" totalImagesPullingTime="22.851453917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:45.294556236 +0000 UTC m=+502.027191274" watchObservedRunningTime="2026-07-24 00:18:45.296690604 +0000 UTC m=+502.029325631" Jul 24 00:18:45.297094 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.297071 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e\": container with ID starting with 85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e not found: ID does not exist" containerID="85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e" Jul 24 00:18:45.297182 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.297101 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e"} err="failed to get container status \"85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e\": rpc error: code = NotFound desc = could not find container \"85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e\": container with ID starting with 85465c8231b43d75cb2098c335a247d2f21c0458fd96acb404a8c8ab44aef72e not found: ID does not exist" Jul 24 00:18:45.391348 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.391313 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4t89\" (UniqueName: \"kubernetes.io/projected/d8013523-aa05-4626-80c8-0d178a38d456-kube-api-access-v4t89\") pod \"d8013523-aa05-4626-80c8-0d178a38d456\" (UID: \"d8013523-aa05-4626-80c8-0d178a38d456\") " Jul 24 00:18:45.391526 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.391375 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbb5f\" (UniqueName: \"kubernetes.io/projected/21e0541a-acad-4d27-aef8-9ec20350f9b5-kube-api-access-gbb5f\") pod \"21e0541a-acad-4d27-aef8-9ec20350f9b5\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " Jul 24 00:18:45.391526 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.391408 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21e0541a-acad-4d27-aef8-9ec20350f9b5-extensions-socket-volume\") pod \"21e0541a-acad-4d27-aef8-9ec20350f9b5\" (UID: \"21e0541a-acad-4d27-aef8-9ec20350f9b5\") " Jul 24 00:18:45.391924 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.391892 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e0541a-acad-4d27-aef8-9ec20350f9b5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "21e0541a-acad-4d27-aef8-9ec20350f9b5" (UID: "21e0541a-acad-4d27-aef8-9ec20350f9b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:45.393498 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.393470 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e0541a-acad-4d27-aef8-9ec20350f9b5-kube-api-access-gbb5f" (OuterVolumeSpecName: "kube-api-access-gbb5f") pod "21e0541a-acad-4d27-aef8-9ec20350f9b5" (UID: "21e0541a-acad-4d27-aef8-9ec20350f9b5"). InnerVolumeSpecName "kube-api-access-gbb5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:45.393597 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.393495 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8013523-aa05-4626-80c8-0d178a38d456-kube-api-access-v4t89" (OuterVolumeSpecName: "kube-api-access-v4t89") pod "d8013523-aa05-4626-80c8-0d178a38d456" (UID: "d8013523-aa05-4626-80c8-0d178a38d456"). InnerVolumeSpecName "kube-api-access-v4t89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:45.492252 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.492220 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4t89\" (UniqueName: \"kubernetes.io/projected/d8013523-aa05-4626-80c8-0d178a38d456-kube-api-access-v4t89\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:45.492252 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.492248 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbb5f\" (UniqueName: \"kubernetes.io/projected/21e0541a-acad-4d27-aef8-9ec20350f9b5-kube-api-access-gbb5f\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:45.492252 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.492258 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21e0541a-acad-4d27-aef8-9ec20350f9b5-extensions-socket-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:45.590680 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.590644 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.592734 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.592707 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.594669 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.594645 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-42ghw\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-42ghw" Jul 24 00:18:45.596668 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:45.596647 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lrwxj\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="d8013523-aa05-4626-80c8-0d178a38d456" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lrwxj" Jul 24 00:18:45.800918 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.800881 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e0541a-acad-4d27-aef8-9ec20350f9b5" path="/var/lib/kubelet/pods/21e0541a-acad-4d27-aef8-9ec20350f9b5/volumes" Jul 24 00:18:45.801278 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:45.801263 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8013523-aa05-4626-80c8-0d178a38d456" path="/var/lib/kubelet/pods/d8013523-aa05-4626-80c8-0d178a38d456/volumes" Jul 24 00:18:46.288690 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.288655 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" event={"ID":"ac84f105-1bad-4085-9b0f-e0c2bed9b20e","Type":"ContainerStarted","Data":"0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa"} Jul 24 00:18:46.289141 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.288705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" event={"ID":"ac84f105-1bad-4085-9b0f-e0c2bed9b20e","Type":"ContainerStarted","Data":"4b3ee4dbb7b244b769c1f532dcdc811f1869bcb3a24e8a6746ae1e7c68eca725"} Jul 24 00:18:46.289141 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.288777 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:46.290445 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.290421 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" event={"ID":"5e71c577-9dd8-4ead-8d07-6d3fabe9e148","Type":"ContainerStarted","Data":"b0ea752514b7018b299cc5a6c71f74556bc462d9b575f213849392b2db8226af"} Jul 24 00:18:46.290579 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.290514 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:46.307461 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.307414 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" podStartSLOduration=15.307400147 podStartE2EDuration="15.307400147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:46.30579805 +0000 UTC m=+503.038433100" watchObservedRunningTime="2026-07-24 00:18:46.307400147 +0000 UTC m=+503.040035171" Jul 24 00:18:46.321538 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:46.321431 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" podStartSLOduration=15.321413559 podStartE2EDuration="15.321413559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:46.32090913 +0000 UTC m=+503.053544151" watchObservedRunningTime="2026-07-24 00:18:46.321413559 +0000 UTC m=+503.054048596" Jul 24 00:18:47.610376 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:47.610330 2585 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-p9jtg_kuadrant-system_3c70127b-108f-44f4-bc77-0bf30b56dfa9_0(1600fc2cda16f2105077bc1939c3019a1de2a668cc49d5460eb6b7e1d782d5d7): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-p9jtg to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 24 00:18:47.610660 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:47.610408 2585 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-p9jtg_kuadrant-system_3c70127b-108f-44f4-bc77-0bf30b56dfa9_0(1600fc2cda16f2105077bc1939c3019a1de2a668cc49d5460eb6b7e1d782d5d7): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-p9jtg to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:48.300489 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.300458 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:48.302789 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:48.302758 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="3c70127b-108f-44f4-bc77-0bf30b56dfa9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:48.314327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.314305 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:48.316507 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:48.316482 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="3c70127b-108f-44f4-bc77-0bf30b56dfa9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:48.420708 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.420676 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j9cd\" (UniqueName: \"kubernetes.io/projected/3c70127b-108f-44f4-bc77-0bf30b56dfa9-kube-api-access-5j9cd\") pod \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " Jul 24 00:18:48.420889 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.420792 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c70127b-108f-44f4-bc77-0bf30b56dfa9-extensions-socket-volume\") pod \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\" (UID: \"3c70127b-108f-44f4-bc77-0bf30b56dfa9\") " Jul 24 00:18:48.421091 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.421067 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70127b-108f-44f4-bc77-0bf30b56dfa9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3c70127b-108f-44f4-bc77-0bf30b56dfa9" (UID: "3c70127b-108f-44f4-bc77-0bf30b56dfa9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:48.422884 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.422846 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c70127b-108f-44f4-bc77-0bf30b56dfa9-kube-api-access-5j9cd" (OuterVolumeSpecName: "kube-api-access-5j9cd") pod "3c70127b-108f-44f4-bc77-0bf30b56dfa9" (UID: "3c70127b-108f-44f4-bc77-0bf30b56dfa9"). InnerVolumeSpecName "kube-api-access-5j9cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:48.521927 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.521892 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c70127b-108f-44f4-bc77-0bf30b56dfa9-extensions-socket-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:48.521927 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:48.521920 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j9cd\" (UniqueName: \"kubernetes.io/projected/3c70127b-108f-44f4-bc77-0bf30b56dfa9-kube-api-access-5j9cd\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:49.305813 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:49.305776 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:49.308210 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:49.308175 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="3c70127b-108f-44f4-bc77-0bf30b56dfa9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:49.313857 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:49.313831 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-p9jtg\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="3c70127b-108f-44f4-bc77-0bf30b56dfa9" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-p9jtg" Jul 24 00:18:49.799978 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:49.799902 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c70127b-108f-44f4-bc77-0bf30b56dfa9" path="/var/lib/kubelet/pods/3c70127b-108f-44f4-bc77-0bf30b56dfa9/volumes" Jul 24 00:18:57.297223 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.297187 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:18:57.297692 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.297368 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:57.366054 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.366017 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc"] Jul 24 00:18:57.366269 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.366222 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" podUID="ac84f105-1bad-4085-9b0f-e0c2bed9b20e" containerName="manager" containerID="cri-o://0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa" gracePeriod=10 Jul 24 00:18:57.572690 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.572658 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm"] Jul 24 00:18:57.573079 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.573065 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8013523-aa05-4626-80c8-0d178a38d456" containerName="manager" Jul 24 00:18:57.573138 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.573080 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8013523-aa05-4626-80c8-0d178a38d456" containerName="manager" Jul 24 00:18:57.573218 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.573190 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8013523-aa05-4626-80c8-0d178a38d456" containerName="manager" Jul 24 00:18:57.575513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.575493 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.589341 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.589318 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm"] Jul 24 00:18:57.611113 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.611087 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42z59\" (UniqueName: \"kubernetes.io/projected/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-kube-api-access-42z59\") pod \"kuadrant-operator-controller-manager-86878c769f-bfwpm\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.611240 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.611137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bfwpm\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.616543 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.616524 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:57.711706 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.711676 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67wkl\" (UniqueName: \"kubernetes.io/projected/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-kube-api-access-67wkl\") pod \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " Jul 24 00:18:57.711894 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.711727 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-extensions-socket-volume\") pod \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\" (UID: \"ac84f105-1bad-4085-9b0f-e0c2bed9b20e\") " Jul 24 00:18:57.711955 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.711917 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42z59\" (UniqueName: \"kubernetes.io/projected/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-kube-api-access-42z59\") pod \"kuadrant-operator-controller-manager-86878c769f-bfwpm\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.711999 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.711986 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bfwpm\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.712233 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.712199 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ac84f105-1bad-4085-9b0f-e0c2bed9b20e" (UID: "ac84f105-1bad-4085-9b0f-e0c2bed9b20e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:18:57.712349 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.712332 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bfwpm\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.713817 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.713787 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-kube-api-access-67wkl" (OuterVolumeSpecName: "kube-api-access-67wkl") pod "ac84f105-1bad-4085-9b0f-e0c2bed9b20e" (UID: "ac84f105-1bad-4085-9b0f-e0c2bed9b20e"). InnerVolumeSpecName "kube-api-access-67wkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:18:57.725217 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.725192 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42z59\" (UniqueName: \"kubernetes.io/projected/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-kube-api-access-42z59\") pod \"kuadrant-operator-controller-manager-86878c769f-bfwpm\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:57.812750 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.812728 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wkl\" (UniqueName: \"kubernetes.io/projected/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-kube-api-access-67wkl\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:57.812750 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.812750 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac84f105-1bad-4085-9b0f-e0c2bed9b20e-extensions-socket-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:18:57.887845 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:57.887767 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:58.020207 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.020183 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm"] Jul 24 00:18:58.022102 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:18:58.022073 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf58ce2_c8f1_4ab2_ab73_efe5fac3faab.slice/crio-69af87ae48e2225c33e34972b7282b68b84371245dae0511a9a67e16a8312224 WatchSource:0}: Error finding container 69af87ae48e2225c33e34972b7282b68b84371245dae0511a9a67e16a8312224: Status 404 returned error can't find the container with id 69af87ae48e2225c33e34972b7282b68b84371245dae0511a9a67e16a8312224 Jul 24 00:18:58.356061 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.356024 2585 generic.go:356] "Generic (PLEG): container finished" podID="ac84f105-1bad-4085-9b0f-e0c2bed9b20e" containerID="0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa" exitCode=0 Jul 24 00:18:58.356477 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.356062 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" event={"ID":"ac84f105-1bad-4085-9b0f-e0c2bed9b20e","Type":"ContainerDied","Data":"0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa"} Jul 24 00:18:58.356477 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.356120 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" event={"ID":"ac84f105-1bad-4085-9b0f-e0c2bed9b20e","Type":"ContainerDied","Data":"4b3ee4dbb7b244b769c1f532dcdc811f1869bcb3a24e8a6746ae1e7c68eca725"} Jul 24 00:18:58.356477 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.356141 2585 scope.go:117] "RemoveContainer" containerID="0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa" Jul 24 00:18:58.356477 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.356071 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc" Jul 24 00:18:58.358079 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.358057 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" event={"ID":"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab","Type":"ContainerStarted","Data":"07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6"} Jul 24 00:18:58.358252 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.358225 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:18:58.358327 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.358258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" event={"ID":"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab","Type":"ContainerStarted","Data":"69af87ae48e2225c33e34972b7282b68b84371245dae0511a9a67e16a8312224"} Jul 24 00:18:58.365446 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.365419 2585 scope.go:117] "RemoveContainer" containerID="0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa" Jul 24 00:18:58.365689 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:18:58.365672 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa\": container with ID starting with 0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa not found: ID does not exist" containerID="0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa" Jul 24 00:18:58.365735 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.365698 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa"} err="failed to get container status \"0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa\": rpc error: code = NotFound desc = could not find container \"0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa\": container with ID starting with 0e3ee292061a5caa5d04d466ae0c3f02ba782a35beaa0441e34590d6e7de8efa not found: ID does not exist" Jul 24 00:18:58.377965 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.377931 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc"] Jul 24 00:18:58.380711 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.380685 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nfvpc"] Jul 24 00:18:58.396416 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:58.396369 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" podStartSLOduration=1.396355826 podStartE2EDuration="1.396355826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:18:58.394936336 +0000 UTC m=+515.127571360" watchObservedRunningTime="2026-07-24 00:18:58.396355826 +0000 UTC m=+515.128990851" Jul 24 00:18:59.799108 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:18:59.799077 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac84f105-1bad-4085-9b0f-e0c2bed9b20e" path="/var/lib/kubelet/pods/ac84f105-1bad-4085-9b0f-e0c2bed9b20e/volumes" Jul 24 00:19:09.367017 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:09.366983 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:19:09.416311 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:09.416248 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5"] Jul 24 00:19:09.416558 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:09.416512 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" podUID="5e71c577-9dd8-4ead-8d07-6d3fabe9e148" containerName="manager" containerID="cri-o://b0ea752514b7018b299cc5a6c71f74556bc462d9b575f213849392b2db8226af" gracePeriod=10 Jul 24 00:19:10.430109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.430079 2585 generic.go:356] "Generic (PLEG): container finished" podID="5e71c577-9dd8-4ead-8d07-6d3fabe9e148" containerID="b0ea752514b7018b299cc5a6c71f74556bc462d9b575f213849392b2db8226af" exitCode=0 Jul 24 00:19:10.430427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.430124 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" event={"ID":"5e71c577-9dd8-4ead-8d07-6d3fabe9e148","Type":"ContainerDied","Data":"b0ea752514b7018b299cc5a6c71f74556bc462d9b575f213849392b2db8226af"} Jul 24 00:19:10.430427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.430176 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" event={"ID":"5e71c577-9dd8-4ead-8d07-6d3fabe9e148","Type":"ContainerDied","Data":"9aff38024162d671b9b0ea3addc4fa9e910ba729ba304cab51f1f91791cf9fcf"} Jul 24 00:19:10.430427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.430192 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aff38024162d671b9b0ea3addc4fa9e910ba729ba304cab51f1f91791cf9fcf" Jul 24 00:19:10.481675 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.481646 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:19:10.519660 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.519635 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbxd2\" (UniqueName: \"kubernetes.io/projected/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-kube-api-access-lbxd2\") pod \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " Jul 24 00:19:10.519842 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.519729 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-extensions-socket-volume\") pod \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\" (UID: \"5e71c577-9dd8-4ead-8d07-6d3fabe9e148\") " Jul 24 00:19:10.520127 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.520101 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5e71c577-9dd8-4ead-8d07-6d3fabe9e148" (UID: "5e71c577-9dd8-4ead-8d07-6d3fabe9e148"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:19:10.521651 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.521630 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-kube-api-access-lbxd2" (OuterVolumeSpecName: "kube-api-access-lbxd2") pod "5e71c577-9dd8-4ead-8d07-6d3fabe9e148" (UID: "5e71c577-9dd8-4ead-8d07-6d3fabe9e148"). InnerVolumeSpecName "kube-api-access-lbxd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:19:10.621203 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.621157 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-extensions-socket-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:19:10.621203 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:10.621196 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxd2\" (UniqueName: \"kubernetes.io/projected/5e71c577-9dd8-4ead-8d07-6d3fabe9e148-kube-api-access-lbxd2\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:19:11.435366 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:11.435333 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5" Jul 24 00:19:11.457086 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:11.457054 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5"] Jul 24 00:19:11.461160 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:11.461136 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rb2m5"] Jul 24 00:19:11.803096 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:11.800679 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e71c577-9dd8-4ead-8d07-6d3fabe9e148" path="/var/lib/kubelet/pods/5e71c577-9dd8-4ead-8d07-6d3fabe9e148/volumes" Jul 24 00:19:25.693806 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.693770 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:25.694323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.694193 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e71c577-9dd8-4ead-8d07-6d3fabe9e148" containerName="manager" Jul 24 00:19:25.694323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.694206 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e71c577-9dd8-4ead-8d07-6d3fabe9e148" containerName="manager" Jul 24 00:19:25.694323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.694217 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac84f105-1bad-4085-9b0f-e0c2bed9b20e" containerName="manager" Jul 24 00:19:25.694323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.694223 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac84f105-1bad-4085-9b0f-e0c2bed9b20e" containerName="manager" Jul 24 00:19:25.694323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.694287 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac84f105-1bad-4085-9b0f-e0c2bed9b20e" containerName="manager" Jul 24 00:19:25.694323 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.694294 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e71c577-9dd8-4ead-8d07-6d3fabe9e148" containerName="manager" Jul 24 00:19:25.704001 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.703974 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:25.704408 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.704376 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:25.706486 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.706465 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 00:19:25.804267 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.804226 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:25.851109 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.851075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88436107-9a9c-44b4-bf32-579d95968f91-config-file\") pod \"limitador-limitador-64c94fd64f-spblp\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:25.851285 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.851125 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbjt\" (UniqueName: \"kubernetes.io/projected/88436107-9a9c-44b4-bf32-579d95968f91-kube-api-access-cfbjt\") pod \"limitador-limitador-64c94fd64f-spblp\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:25.952606 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.952505 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbjt\" (UniqueName: \"kubernetes.io/projected/88436107-9a9c-44b4-bf32-579d95968f91-kube-api-access-cfbjt\") pod \"limitador-limitador-64c94fd64f-spblp\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:25.952844 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.952821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88436107-9a9c-44b4-bf32-579d95968f91-config-file\") pod \"limitador-limitador-64c94fd64f-spblp\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:25.953488 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.953467 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88436107-9a9c-44b4-bf32-579d95968f91-config-file\") pod \"limitador-limitador-64c94fd64f-spblp\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:25.960064 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:25.960038 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbjt\" (UniqueName: \"kubernetes.io/projected/88436107-9a9c-44b4-bf32-579d95968f91-kube-api-access-cfbjt\") pod \"limitador-limitador-64c94fd64f-spblp\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:26.015685 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.015635 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:26.153113 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.153083 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:26.154954 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:19:26.154930 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88436107_9a9c_44b4_bf32_579d95968f91.slice/crio-b1aa46752681f7c96207ba5b0f3da301e9bb22d08efe64169b6dba7004b0e243 WatchSource:0}: Error finding container b1aa46752681f7c96207ba5b0f3da301e9bb22d08efe64169b6dba7004b0e243: Status 404 returned error can't find the container with id b1aa46752681f7c96207ba5b0f3da301e9bb22d08efe64169b6dba7004b0e243 Jul 24 00:19:26.176262 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.176237 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:19:26.179985 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.179965 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.188318 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.188295 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:19:26.213311 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.213232 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:19:26.256210 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.256170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvpqg\" (UniqueName: \"kubernetes.io/projected/3c81af3b-ef11-4850-9d1a-39911fe34aba-kube-api-access-tvpqg\") pod \"limitador-limitador-54d6d988d5-589ln\" (UID: \"3c81af3b-ef11-4850-9d1a-39911fe34aba\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.256363 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.256299 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c81af3b-ef11-4850-9d1a-39911fe34aba-config-file\") pod \"limitador-limitador-54d6d988d5-589ln\" (UID: \"3c81af3b-ef11-4850-9d1a-39911fe34aba\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.357174 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.357138 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpqg\" (UniqueName: \"kubernetes.io/projected/3c81af3b-ef11-4850-9d1a-39911fe34aba-kube-api-access-tvpqg\") pod \"limitador-limitador-54d6d988d5-589ln\" (UID: \"3c81af3b-ef11-4850-9d1a-39911fe34aba\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.357368 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.357201 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c81af3b-ef11-4850-9d1a-39911fe34aba-config-file\") pod \"limitador-limitador-54d6d988d5-589ln\" (UID: \"3c81af3b-ef11-4850-9d1a-39911fe34aba\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.357799 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.357780 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c81af3b-ef11-4850-9d1a-39911fe34aba-config-file\") pod \"limitador-limitador-54d6d988d5-589ln\" (UID: \"3c81af3b-ef11-4850-9d1a-39911fe34aba\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.364485 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.364452 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpqg\" (UniqueName: \"kubernetes.io/projected/3c81af3b-ef11-4850-9d1a-39911fe34aba-kube-api-access-tvpqg\") pod \"limitador-limitador-54d6d988d5-589ln\" (UID: \"3c81af3b-ef11-4850-9d1a-39911fe34aba\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.491457 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.491378 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:26.522946 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.522913 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" event={"ID":"88436107-9a9c-44b4-bf32-579d95968f91","Type":"ContainerStarted","Data":"b1aa46752681f7c96207ba5b0f3da301e9bb22d08efe64169b6dba7004b0e243"} Jul 24 00:19:26.627527 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:26.627503 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:19:26.629447 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:19:26.629421 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c81af3b_ef11_4850_9d1a_39911fe34aba.slice/crio-8964b50d963682b408b0cdedfd872c4f8f327cfc0e7df929b4602b0ce5be4ea2 WatchSource:0}: Error finding container 8964b50d963682b408b0cdedfd872c4f8f327cfc0e7df929b4602b0ce5be4ea2: Status 404 returned error can't find the container with id 8964b50d963682b408b0cdedfd872c4f8f327cfc0e7df929b4602b0ce5be4ea2 Jul 24 00:19:27.536900 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:27.536809 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" event={"ID":"3c81af3b-ef11-4850-9d1a-39911fe34aba","Type":"ContainerStarted","Data":"8964b50d963682b408b0cdedfd872c4f8f327cfc0e7df929b4602b0ce5be4ea2"} Jul 24 00:19:29.558151 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:29.558117 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" event={"ID":"3c81af3b-ef11-4850-9d1a-39911fe34aba","Type":"ContainerStarted","Data":"71dbdc042d26c9c867d7596c2d6de50f2a75c53cefe0f47655fb29589aecc631"} Jul 24 00:19:29.558570 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:29.558208 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:29.559712 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:29.559688 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" event={"ID":"88436107-9a9c-44b4-bf32-579d95968f91","Type":"ContainerStarted","Data":"efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3"} Jul 24 00:19:29.559831 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:29.559819 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:29.575010 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:29.574962 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" podStartSLOduration=1.010786347 podStartE2EDuration="3.574946931s" totalImagesPullingTime="2.564160584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:29.572920525 +0000 UTC m=+546.305555550" watchObservedRunningTime="2026-07-24 00:19:29.574946931 +0000 UTC m=+546.307581977" Jul 24 00:19:29.588307 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:29.588269 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" podStartSLOduration=1.557171621 podStartE2EDuration="4.588254804s" totalImagesPullingTime="3.031083183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:29.586388342 +0000 UTC m=+546.319023378" watchObservedRunningTime="2026-07-24 00:19:29.588254804 +0000 UTC m=+546.320889829" Jul 24 00:19:40.566195 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:40.566162 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-589ln" Jul 24 00:19:40.566735 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:40.566317 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:40.626202 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:40.626162 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:40.626744 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:40.626690 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" podUID="88436107-9a9c-44b4-bf32-579d95968f91" containerName="limitador" containerID="cri-o://efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3" gracePeriod=30 Jul 24 00:19:41.188029 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.187964 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:41.286413 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.286381 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfbjt\" (UniqueName: \"kubernetes.io/projected/88436107-9a9c-44b4-bf32-579d95968f91-kube-api-access-cfbjt\") pod \"88436107-9a9c-44b4-bf32-579d95968f91\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " Jul 24 00:19:41.286588 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.286433 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88436107-9a9c-44b4-bf32-579d95968f91-config-file\") pod \"88436107-9a9c-44b4-bf32-579d95968f91\" (UID: \"88436107-9a9c-44b4-bf32-579d95968f91\") " Jul 24 00:19:41.286821 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.286784 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88436107-9a9c-44b4-bf32-579d95968f91-config-file" (OuterVolumeSpecName: "config-file") pod "88436107-9a9c-44b4-bf32-579d95968f91" (UID: "88436107-9a9c-44b4-bf32-579d95968f91"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:19:41.288430 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.288407 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88436107-9a9c-44b4-bf32-579d95968f91-kube-api-access-cfbjt" (OuterVolumeSpecName: "kube-api-access-cfbjt") pod "88436107-9a9c-44b4-bf32-579d95968f91" (UID: "88436107-9a9c-44b4-bf32-579d95968f91"). InnerVolumeSpecName "kube-api-access-cfbjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:19:41.388032 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.387997 2585 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/88436107-9a9c-44b4-bf32-579d95968f91-config-file\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:19:41.388032 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.388028 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbjt\" (UniqueName: \"kubernetes.io/projected/88436107-9a9c-44b4-bf32-579d95968f91-kube-api-access-cfbjt\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:19:41.626872 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.626839 2585 generic.go:356] "Generic (PLEG): container finished" podID="88436107-9a9c-44b4-bf32-579d95968f91" containerID="efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3" exitCode=0 Jul 24 00:19:41.627352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.626905 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" Jul 24 00:19:41.627352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.626931 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" event={"ID":"88436107-9a9c-44b4-bf32-579d95968f91","Type":"ContainerDied","Data":"efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3"} Jul 24 00:19:41.627352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.627000 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-spblp" event={"ID":"88436107-9a9c-44b4-bf32-579d95968f91","Type":"ContainerDied","Data":"b1aa46752681f7c96207ba5b0f3da301e9bb22d08efe64169b6dba7004b0e243"} Jul 24 00:19:41.627352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.627035 2585 scope.go:117] "RemoveContainer" containerID="efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3" Jul 24 00:19:41.636824 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.636807 2585 scope.go:117] "RemoveContainer" containerID="efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3" Jul 24 00:19:41.637149 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:19:41.637132 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3\": container with ID starting with efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3 not found: ID does not exist" containerID="efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3" Jul 24 00:19:41.637207 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.637157 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3"} err="failed to get container status \"efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3\": rpc error: code = NotFound desc = could not find container \"efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3\": container with ID starting with efc921e96201fc6447fd5d6a9bf8dffe541a932f52dcddc2ac40b5e170df55f3 not found: ID does not exist" Jul 24 00:19:41.649378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.649358 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:41.654035 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.654013 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-spblp"] Jul 24 00:19:41.799297 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:41.799266 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88436107-9a9c-44b4-bf32-579d95968f91" path="/var/lib/kubelet/pods/88436107-9a9c-44b4-bf32-579d95968f91/volumes" Jul 24 00:19:42.658783 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.658749 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8mj77"] Jul 24 00:19:42.659292 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.659160 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88436107-9a9c-44b4-bf32-579d95968f91" containerName="limitador" Jul 24 00:19:42.659292 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.659173 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="88436107-9a9c-44b4-bf32-579d95968f91" containerName="limitador" Jul 24 00:19:42.659292 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.659267 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="88436107-9a9c-44b4-bf32-579d95968f91" containerName="limitador" Jul 24 00:19:42.663779 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.663758 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.666360 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.666338 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 00:19:42.666485 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.666351 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 00:19:42.667642 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.667609 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lqh87\"" Jul 24 00:19:42.667749 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.667646 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 00:19:42.670560 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.670535 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8mj77"] Jul 24 00:19:42.800227 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.800196 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxq4m\" (UniqueName: \"kubernetes.io/projected/6ff37e38-0287-439f-9859-b989e4613269-kube-api-access-kxq4m\") pod \"postgres-69fb4594cf-8mj77\" (UID: \"6ff37e38-0287-439f-9859-b989e4613269\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.800406 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.800271 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6ff37e38-0287-439f-9859-b989e4613269-data\") pod \"postgres-69fb4594cf-8mj77\" (UID: \"6ff37e38-0287-439f-9859-b989e4613269\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.901204 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.901170 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6ff37e38-0287-439f-9859-b989e4613269-data\") pod \"postgres-69fb4594cf-8mj77\" (UID: \"6ff37e38-0287-439f-9859-b989e4613269\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.901407 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.901281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxq4m\" (UniqueName: \"kubernetes.io/projected/6ff37e38-0287-439f-9859-b989e4613269-kube-api-access-kxq4m\") pod \"postgres-69fb4594cf-8mj77\" (UID: \"6ff37e38-0287-439f-9859-b989e4613269\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.901629 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.901605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6ff37e38-0287-439f-9859-b989e4613269-data\") pod \"postgres-69fb4594cf-8mj77\" (UID: \"6ff37e38-0287-439f-9859-b989e4613269\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.909120 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.909059 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxq4m\" (UniqueName: \"kubernetes.io/projected/6ff37e38-0287-439f-9859-b989e4613269-kube-api-access-kxq4m\") pod \"postgres-69fb4594cf-8mj77\" (UID: \"6ff37e38-0287-439f-9859-b989e4613269\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:42.975123 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:42.975085 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:43.105672 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:43.105641 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8mj77"] Jul 24 00:19:43.107112 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:19:43.107079 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff37e38_0287_439f_9859_b989e4613269.slice/crio-d89f7a8be8514f46a755f6fb5c156d0d1b935417b9856853238048244232434b WatchSource:0}: Error finding container d89f7a8be8514f46a755f6fb5c156d0d1b935417b9856853238048244232434b: Status 404 returned error can't find the container with id d89f7a8be8514f46a755f6fb5c156d0d1b935417b9856853238048244232434b Jul 24 00:19:43.639361 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:43.639326 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" event={"ID":"6ff37e38-0287-439f-9859-b989e4613269","Type":"ContainerStarted","Data":"d89f7a8be8514f46a755f6fb5c156d0d1b935417b9856853238048244232434b"} Jul 24 00:19:48.674790 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:48.674746 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" event={"ID":"6ff37e38-0287-439f-9859-b989e4613269","Type":"ContainerStarted","Data":"14d60811527c68d0e69665352577a5dfbf122dad82076e00a5f98b428dcecc9f"} Jul 24 00:19:48.675177 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:48.674826 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:19:48.691162 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:48.691110 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" podStartSLOduration=1.281636218 podStartE2EDuration="6.691092329s" totalImagesPullingTime="5.409456111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:19:48.689467119 +0000 UTC m=+565.422102148" watchObservedRunningTime="2026-07-24 00:19:48.691092329 +0000 UTC m=+565.423727353" Jul 24 00:19:54.709879 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:19:54.709838 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8mj77" Jul 24 00:20:01.703995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.703963 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8497945f8d-tbswc"] Jul 24 00:20:01.860998 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.860959 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8497945f8d-tbswc"] Jul 24 00:20:01.861204 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.861087 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:01.864298 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.864267 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 00:20:01.864459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.864431 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2njs6\"" Jul 24 00:20:01.864703 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.864687 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 00:20:01.878902 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.878856 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-9mr4z"] Jul 24 00:20:01.904504 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.904478 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-9mr4z"] Jul 24 00:20:01.904631 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.904572 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:01.970064 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.969977 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmj6l\" (UniqueName: \"kubernetes.io/projected/a0416b9b-f53f-4c35-ae59-61750a138fbd-kube-api-access-mmj6l\") pod \"maas-controller-8497945f8d-tbswc\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:01.970064 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.970015 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/003a5fb2-619a-4c83-be53-7a825cc9b373-webhook-cert\") pod \"maas-controller-5b9bbd68d9-9mr4z\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:01.970276 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.970114 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjsm\" (UniqueName: \"kubernetes.io/projected/003a5fb2-619a-4c83-be53-7a825cc9b373-kube-api-access-rvjsm\") pod \"maas-controller-5b9bbd68d9-9mr4z\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:01.970276 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:01.970170 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0416b9b-f53f-4c35-ae59-61750a138fbd-webhook-cert\") pod \"maas-controller-8497945f8d-tbswc\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:02.070961 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.070925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjsm\" (UniqueName: \"kubernetes.io/projected/003a5fb2-619a-4c83-be53-7a825cc9b373-kube-api-access-rvjsm\") pod \"maas-controller-5b9bbd68d9-9mr4z\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:02.071169 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.070989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0416b9b-f53f-4c35-ae59-61750a138fbd-webhook-cert\") pod \"maas-controller-8497945f8d-tbswc\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:02.071169 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.071017 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmj6l\" (UniqueName: \"kubernetes.io/projected/a0416b9b-f53f-4c35-ae59-61750a138fbd-kube-api-access-mmj6l\") pod \"maas-controller-8497945f8d-tbswc\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:02.071169 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.071038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/003a5fb2-619a-4c83-be53-7a825cc9b373-webhook-cert\") pod \"maas-controller-5b9bbd68d9-9mr4z\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:02.073395 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.073368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/003a5fb2-619a-4c83-be53-7a825cc9b373-webhook-cert\") pod \"maas-controller-5b9bbd68d9-9mr4z\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:02.073499 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.073474 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0416b9b-f53f-4c35-ae59-61750a138fbd-webhook-cert\") pod \"maas-controller-8497945f8d-tbswc\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:02.079349 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.079318 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmj6l\" (UniqueName: \"kubernetes.io/projected/a0416b9b-f53f-4c35-ae59-61750a138fbd-kube-api-access-mmj6l\") pod \"maas-controller-8497945f8d-tbswc\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:02.079349 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.079328 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjsm\" (UniqueName: \"kubernetes.io/projected/003a5fb2-619a-4c83-be53-7a825cc9b373-kube-api-access-rvjsm\") pod \"maas-controller-5b9bbd68d9-9mr4z\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:02.172693 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.172657 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:02.214829 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.214792 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:02.320053 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.320029 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8497945f8d-tbswc"] Jul 24 00:20:02.321533 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:20:02.321511 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0416b9b_f53f_4c35_ae59_61750a138fbd.slice/crio-9364fb06048707b4a43b49795a7aeb6cd76309191e234ea47acacc3fb121a490 WatchSource:0}: Error finding container 9364fb06048707b4a43b49795a7aeb6cd76309191e234ea47acacc3fb121a490: Status 404 returned error can't find the container with id 9364fb06048707b4a43b49795a7aeb6cd76309191e234ea47acacc3fb121a490 Jul 24 00:20:02.381958 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.381930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-9mr4z"] Jul 24 00:20:02.383138 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:20:02.383115 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003a5fb2_619a_4c83_be53_7a825cc9b373.slice/crio-c9c60455378d8a96893314662e10a726622c746c3f313c6262afb80876d64517 WatchSource:0}: Error finding container c9c60455378d8a96893314662e10a726622c746c3f313c6262afb80876d64517: Status 404 returned error can't find the container with id c9c60455378d8a96893314662e10a726622c746c3f313c6262afb80876d64517 Jul 24 00:20:02.753666 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.753628 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" event={"ID":"003a5fb2-619a-4c83-be53-7a825cc9b373","Type":"ContainerStarted","Data":"c9c60455378d8a96893314662e10a726622c746c3f313c6262afb80876d64517"} Jul 24 00:20:02.754782 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:02.754756 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8497945f8d-tbswc" event={"ID":"a0416b9b-f53f-4c35-ae59-61750a138fbd","Type":"ContainerStarted","Data":"9364fb06048707b4a43b49795a7aeb6cd76309191e234ea47acacc3fb121a490"} Jul 24 00:20:05.775970 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:05.775840 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" event={"ID":"003a5fb2-619a-4c83-be53-7a825cc9b373","Type":"ContainerStarted","Data":"2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581"} Jul 24 00:20:05.776403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:05.775982 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:05.777337 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:05.777313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8497945f8d-tbswc" event={"ID":"a0416b9b-f53f-4c35-ae59-61750a138fbd","Type":"ContainerStarted","Data":"07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88"} Jul 24 00:20:05.777461 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:05.777445 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:05.795204 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:05.795156 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" podStartSLOduration=1.871645983 podStartE2EDuration="4.795139999s" totalImagesPullingTime="2.923494016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:05.793507312 +0000 UTC m=+582.526142338" watchObservedRunningTime="2026-07-24 00:20:05.795139999 +0000 UTC m=+582.527775024" Jul 24 00:20:05.812881 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:05.812811 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8497945f8d-tbswc" podStartSLOduration=1.817517623 podStartE2EDuration="4.812795391s" totalImagesPullingTime="2.995277768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:05.811086429 +0000 UTC m=+582.543721455" watchObservedRunningTime="2026-07-24 00:20:05.812795391 +0000 UTC m=+582.545430415" Jul 24 00:20:16.785308 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:16.785275 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:16.785901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:16.785470 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:20:16.840407 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:16.840375 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8497945f8d-tbswc"] Jul 24 00:20:16.840644 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:16.840617 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8497945f8d-tbswc" podUID="a0416b9b-f53f-4c35-ae59-61750a138fbd" containerName="manager" containerID="cri-o://07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88" gracePeriod=30 Jul 24 00:20:17.080940 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.080918 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:17.205318 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.205282 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0416b9b-f53f-4c35-ae59-61750a138fbd-webhook-cert\") pod \"a0416b9b-f53f-4c35-ae59-61750a138fbd\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " Jul 24 00:20:17.205318 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.205323 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmj6l\" (UniqueName: \"kubernetes.io/projected/a0416b9b-f53f-4c35-ae59-61750a138fbd-kube-api-access-mmj6l\") pod \"a0416b9b-f53f-4c35-ae59-61750a138fbd\" (UID: \"a0416b9b-f53f-4c35-ae59-61750a138fbd\") " Jul 24 00:20:17.207608 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.207572 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0416b9b-f53f-4c35-ae59-61750a138fbd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a0416b9b-f53f-4c35-ae59-61750a138fbd" (UID: "a0416b9b-f53f-4c35-ae59-61750a138fbd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:20:17.207722 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.207678 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0416b9b-f53f-4c35-ae59-61750a138fbd-kube-api-access-mmj6l" (OuterVolumeSpecName: "kube-api-access-mmj6l") pod "a0416b9b-f53f-4c35-ae59-61750a138fbd" (UID: "a0416b9b-f53f-4c35-ae59-61750a138fbd"). InnerVolumeSpecName "kube-api-access-mmj6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:20:17.306088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.306054 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0416b9b-f53f-4c35-ae59-61750a138fbd-webhook-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:20:17.306088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.306082 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmj6l\" (UniqueName: \"kubernetes.io/projected/a0416b9b-f53f-4c35-ae59-61750a138fbd-kube-api-access-mmj6l\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:20:17.845399 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.845360 2585 generic.go:356] "Generic (PLEG): container finished" podID="a0416b9b-f53f-4c35-ae59-61750a138fbd" containerID="07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88" exitCode=0 Jul 24 00:20:17.845399 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.845399 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8497945f8d-tbswc" Jul 24 00:20:17.845849 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.845451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8497945f8d-tbswc" event={"ID":"a0416b9b-f53f-4c35-ae59-61750a138fbd","Type":"ContainerDied","Data":"07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88"} Jul 24 00:20:17.845849 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.845501 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8497945f8d-tbswc" event={"ID":"a0416b9b-f53f-4c35-ae59-61750a138fbd","Type":"ContainerDied","Data":"9364fb06048707b4a43b49795a7aeb6cd76309191e234ea47acacc3fb121a490"} Jul 24 00:20:17.845849 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.845524 2585 scope.go:117] "RemoveContainer" containerID="07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88" Jul 24 00:20:17.854681 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.854664 2585 scope.go:117] "RemoveContainer" containerID="07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88" Jul 24 00:20:17.854955 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:20:17.854935 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88\": container with ID starting with 07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88 not found: ID does not exist" containerID="07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88" Jul 24 00:20:17.855050 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.854964 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88"} err="failed to get container status \"07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88\": rpc error: code = NotFound desc = could not find container \"07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88\": container with ID starting with 07c3137396b4c91229ff656feabb3f64e8defd5b7069420f5060e02ed87d4f88 not found: ID does not exist" Jul 24 00:20:17.864829 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.864808 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8497945f8d-tbswc"] Jul 24 00:20:17.868198 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:17.868179 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8497945f8d-tbswc"] Jul 24 00:20:19.799989 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:19.799959 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0416b9b-f53f-4c35-ae59-61750a138fbd" path="/var/lib/kubelet/pods/a0416b9b-f53f-4c35-ae59-61750a138fbd/volumes" Jul 24 00:20:23.710663 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:23.710637 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:20:23.711124 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:23.710638 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:20:38.914701 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.914667 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dc745548d-6gmb9"] Jul 24 00:20:38.923975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.915087 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0416b9b-f53f-4c35-ae59-61750a138fbd" containerName="manager" Jul 24 00:20:38.923975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.915100 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0416b9b-f53f-4c35-ae59-61750a138fbd" containerName="manager" Jul 24 00:20:38.923975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.915175 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0416b9b-f53f-4c35-ae59-61750a138fbd" containerName="manager" Jul 24 00:20:38.925671 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.925649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:38.928246 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.928217 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc745548d-6gmb9"] Jul 24 00:20:38.928376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.928352 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nv7dj\"" Jul 24 00:20:38.928431 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:38.928383 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 00:20:39.095147 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.095103 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-openshift-service-ca\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.095357 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.095197 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf2s\" (UniqueName: \"kubernetes.io/projected/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-kube-api-access-9jf2s\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.095357 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.095249 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-maas-api-tls\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.196782 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.196687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf2s\" (UniqueName: \"kubernetes.io/projected/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-kube-api-access-9jf2s\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.196782 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.196747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-maas-api-tls\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.196983 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.196798 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-openshift-service-ca\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.196983 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:20:39.196938 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 24 00:20:39.197055 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:20:39.197012 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-maas-api-tls podName:c2544ac4-8cbc-451a-9fcf-5317fd95bb03 nodeName:}" failed. No retries permitted until 2026-07-24 00:20:39.696991637 +0000 UTC m=+616.429626639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-maas-api-tls") pod "maas-api-dc745548d-6gmb9" (UID: "c2544ac4-8cbc-451a-9fcf-5317fd95bb03") : secret "maas-api-serving-cert" not found Jul 24 00:20:39.197453 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.197434 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-openshift-service-ca\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.204110 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.204080 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf2s\" (UniqueName: \"kubernetes.io/projected/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-kube-api-access-9jf2s\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.418185 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.418142 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5x24f"] Jul 24 00:20:39.422635 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.422613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.425396 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.425372 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 00:20:39.425529 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.425372 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-k7c2z\"" Jul 24 00:20:39.432198 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.432174 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5x24f"] Jul 24 00:20:39.456722 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.456643 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:20:39.600228 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.600198 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262c57a9-2608-4235-bb94-b67ac50ac889-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5x24f\" (UID: \"262c57a9-2608-4235-bb94-b67ac50ac889\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.600408 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.600253 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwr5l\" (UniqueName: \"kubernetes.io/projected/262c57a9-2608-4235-bb94-b67ac50ac889-kube-api-access-zwr5l\") pod \"payload-processing-5cc6cd444b-5x24f\" (UID: \"262c57a9-2608-4235-bb94-b67ac50ac889\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.701605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.701570 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262c57a9-2608-4235-bb94-b67ac50ac889-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5x24f\" (UID: \"262c57a9-2608-4235-bb94-b67ac50ac889\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.701768 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.701614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-maas-api-tls\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.701768 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.701639 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwr5l\" (UniqueName: \"kubernetes.io/projected/262c57a9-2608-4235-bb94-b67ac50ac889-kube-api-access-zwr5l\") pod \"payload-processing-5cc6cd444b-5x24f\" (UID: \"262c57a9-2608-4235-bb94-b67ac50ac889\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.702638 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.702609 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262c57a9-2608-4235-bb94-b67ac50ac889-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-5x24f\" (UID: \"262c57a9-2608-4235-bb94-b67ac50ac889\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.704003 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.703981 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2544ac4-8cbc-451a-9fcf-5317fd95bb03-maas-api-tls\") pod \"maas-api-dc745548d-6gmb9\" (UID: \"c2544ac4-8cbc-451a-9fcf-5317fd95bb03\") " pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.711305 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.711236 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwr5l\" (UniqueName: \"kubernetes.io/projected/262c57a9-2608-4235-bb94-b67ac50ac889-kube-api-access-zwr5l\") pod \"payload-processing-5cc6cd444b-5x24f\" (UID: \"262c57a9-2608-4235-bb94-b67ac50ac889\") " pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.734618 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.734588 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:39.838355 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.838322 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:39.877201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.877166 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-5x24f"] Jul 24 00:20:39.879634 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:20:39.879607 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262c57a9_2608_4235_bb94_b67ac50ac889.slice/crio-427e779b1a69a68f57523c009046681edbd741621fff808cdbb8b4ec5ea00bd5 WatchSource:0}: Error finding container 427e779b1a69a68f57523c009046681edbd741621fff808cdbb8b4ec5ea00bd5: Status 404 returned error can't find the container with id 427e779b1a69a68f57523c009046681edbd741621fff808cdbb8b4ec5ea00bd5 Jul 24 00:20:39.882028 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.882013 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:20:39.980747 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.980645 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" event={"ID":"262c57a9-2608-4235-bb94-b67ac50ac889","Type":"ContainerStarted","Data":"427e779b1a69a68f57523c009046681edbd741621fff808cdbb8b4ec5ea00bd5"} Jul 24 00:20:39.987538 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:39.987503 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dc745548d-6gmb9"] Jul 24 00:20:39.988760 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:20:39.988711 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2544ac4_8cbc_451a_9fcf_5317fd95bb03.slice/crio-8036d638cb3807dc1a93bafb5b5674bc536f7cd0a8c5f99f0f39b6d654169900 WatchSource:0}: Error finding container 8036d638cb3807dc1a93bafb5b5674bc536f7cd0a8c5f99f0f39b6d654169900: Status 404 returned error can't find the container with id 8036d638cb3807dc1a93bafb5b5674bc536f7cd0a8c5f99f0f39b6d654169900 Jul 24 00:20:40.991713 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:40.991628 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" event={"ID":"c2544ac4-8cbc-451a-9fcf-5317fd95bb03","Type":"ContainerStarted","Data":"8036d638cb3807dc1a93bafb5b5674bc536f7cd0a8c5f99f0f39b6d654169900"} Jul 24 00:20:41.999161 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:41.999018 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" event={"ID":"c2544ac4-8cbc-451a-9fcf-5317fd95bb03","Type":"ContainerStarted","Data":"b67a84fe09d01853dbae2759d15b52b9bc08dfb73f9df3a8f409cfd32615118e"} Jul 24 00:20:41.999161 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:41.999085 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:42.000587 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:42.000562 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" event={"ID":"262c57a9-2608-4235-bb94-b67ac50ac889","Type":"ContainerStarted","Data":"836b686300a269249265b112ed4709977490f8fcfc1467dbe8568c918dd94b99"} Jul 24 00:20:42.000725 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:42.000680 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:42.015112 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:42.015057 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" podStartSLOduration=2.178649658 podStartE2EDuration="4.015042139s" totalImagesPullingTime="1.836392481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:42.014170374 +0000 UTC m=+618.746805400" watchObservedRunningTime="2026-07-24 00:20:42.015042139 +0000 UTC m=+618.747677164" Jul 24 00:20:42.029771 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:42.029723 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" podStartSLOduration=1.0797468669999999 podStartE2EDuration="3.029707542s" totalImagesPullingTime="1.949960675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:20:42.027937562 +0000 UTC m=+618.760572589" watchObservedRunningTime="2026-07-24 00:20:42.029707542 +0000 UTC m=+618.762342566" Jul 24 00:20:48.011479 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:48.011401 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dc745548d-6gmb9" Jul 24 00:20:53.007903 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:53.007850 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-5x24f" Jul 24 00:20:59.832904 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.832852 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75"] Jul 24 00:20:59.836637 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.836616 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.839428 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.839402 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 00:20:59.840610 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.840587 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 00:20:59.840805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.840619 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k8tlc\"" Jul 24 00:20:59.840805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.840649 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 00:20:59.845319 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.845291 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75"] Jul 24 00:20:59.870220 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.870188 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.870369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.870230 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.870369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.870298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.870369 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.870323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcq7l\" (UniqueName: \"kubernetes.io/projected/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-kube-api-access-pcq7l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.870501 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.870382 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.870501 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.870462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971122 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcq7l\" (UniqueName: \"kubernetes.io/projected/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-kube-api-access-pcq7l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971329 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971573 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971340 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971573 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.971748 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.971723 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.972408 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.972380 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.974225 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.974197 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.974595 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.974574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:20:59.979223 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:20:59.979200 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcq7l\" (UniqueName: \"kubernetes.io/projected/3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf-kube-api-access-pcq7l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75\" (UID: \"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:21:00.148022 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:00.147929 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:21:00.240257 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:00.239746 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:21:00.291161 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:00.291134 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75"] Jul 24 00:21:00.293380 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:21:00.293350 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcc91f9_06c7_490c_aa7c_43c5c2f11aaf.slice/crio-f39d0a16d0e03146bd747bccfaeb480456ecb14d9093e7e021f81528fea32498 WatchSource:0}: Error finding container f39d0a16d0e03146bd747bccfaeb480456ecb14d9093e7e021f81528fea32498: Status 404 returned error can't find the container with id f39d0a16d0e03146bd747bccfaeb480456ecb14d9093e7e021f81528fea32498 Jul 24 00:21:01.116434 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:01.116391 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" event={"ID":"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf","Type":"ContainerStarted","Data":"f39d0a16d0e03146bd747bccfaeb480456ecb14d9093e7e021f81528fea32498"} Jul 24 00:21:02.124487 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:02.124452 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" event={"ID":"3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf","Type":"ContainerStarted","Data":"e3fdfae1e862a728aef74e9c78978d7b9091b80d909fbaab623f49883c4571bf"} Jul 24 00:21:02.141378 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:02.141329 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" podStartSLOduration=2.043034459 podStartE2EDuration="3.141311999s" totalImagesPullingTime="1.09827754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:02.139409643 +0000 UTC m=+638.872044681" watchObservedRunningTime="2026-07-24 00:21:02.141311999 +0000 UTC m=+638.873947023" Jul 24 00:21:10.148088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:10.148046 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:21:10.148514 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:10.148370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:21:10.161094 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:10.161066 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:21:10.189082 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:10.189052 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75" Jul 24 00:21:21.525185 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:21.525145 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:21:40.631760 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:40.631717 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:21:45.129017 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.128984 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h"] Jul 24 00:21:45.134456 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.134434 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.136995 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.136977 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 00:21:45.141721 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.141696 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h"] Jul 24 00:21:45.183084 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.183047 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9j55\" (UniqueName: \"kubernetes.io/projected/0ce82edb-f2ca-4760-949f-0ba9d9674c55-kube-api-access-h9j55\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.183273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.183088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.183273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.183185 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce82edb-f2ca-4760-949f-0ba9d9674c55-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.183273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.183248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.183383 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.183290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.183383 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.183338 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284018 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.283978 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9j55\" (UniqueName: \"kubernetes.io/projected/0ce82edb-f2ca-4760-949f-0ba9d9674c55-kube-api-access-h9j55\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284018 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284017 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce82edb-f2ca-4760-949f-0ba9d9674c55-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284106 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284232 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284128 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284490 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284464 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284490 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.284639 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.284621 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.286448 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.286425 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ce82edb-f2ca-4760-949f-0ba9d9674c55-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.286723 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.286707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ce82edb-f2ca-4760-949f-0ba9d9674c55-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.291462 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.291442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9j55\" (UniqueName: \"kubernetes.io/projected/0ce82edb-f2ca-4760-949f-0ba9d9674c55-kube-api-access-h9j55\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h\" (UID: \"0ce82edb-f2ca-4760-949f-0ba9d9674c55\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.448031 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.447936 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:45.600220 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:45.600195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h"] Jul 24 00:21:46.380508 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:46.380473 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" event={"ID":"0ce82edb-f2ca-4760-949f-0ba9d9674c55","Type":"ContainerStarted","Data":"4aa6fa662085b5aa7f7946fc2041e0fb8270c13c3ba0a3926a43f3aa159e3f16"} Jul 24 00:21:46.380968 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:46.380532 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" event={"ID":"0ce82edb-f2ca-4760-949f-0ba9d9674c55","Type":"ContainerStarted","Data":"82aee4a91eae213c49c0cc9f56da20d829372c2fbc13bf00d1c33474cc8cfb60"} Jul 24 00:21:46.398680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:46.398620 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" podStartSLOduration=1.190621223 podStartE2EDuration="1.398600975s" totalImagesPullingTime="207.979752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:21:46.395838578 +0000 UTC m=+683.128473625" watchObservedRunningTime="2026-07-24 00:21:46.398600975 +0000 UTC m=+683.131236001" Jul 24 00:21:47.145565 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:47.145526 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:21:54.735477 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:54.735443 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:21:55.448362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:55.448335 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:55.448557 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:55.448394 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:55.460953 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:55.460928 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:21:56.449843 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:21:56.449812 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h" Jul 24 00:22:03.536235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:22:03.536202 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:23:54.032635 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:23:54.032562 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:24:03.825857 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:03.825822 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:24:11.450749 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.450701 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-9mr4z"] Jul 24 00:24:11.451271 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.450961 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" podUID="003a5fb2-619a-4c83-be53-7a825cc9b373" containerName="manager" containerID="cri-o://2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581" gracePeriod=30 Jul 24 00:24:11.712342 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.712283 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:24:11.809586 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.809558 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvjsm\" (UniqueName: \"kubernetes.io/projected/003a5fb2-619a-4c83-be53-7a825cc9b373-kube-api-access-rvjsm\") pod \"003a5fb2-619a-4c83-be53-7a825cc9b373\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " Jul 24 00:24:11.809744 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.809599 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/003a5fb2-619a-4c83-be53-7a825cc9b373-webhook-cert\") pod \"003a5fb2-619a-4c83-be53-7a825cc9b373\" (UID: \"003a5fb2-619a-4c83-be53-7a825cc9b373\") " Jul 24 00:24:11.811596 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.811567 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003a5fb2-619a-4c83-be53-7a825cc9b373-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "003a5fb2-619a-4c83-be53-7a825cc9b373" (UID: "003a5fb2-619a-4c83-be53-7a825cc9b373"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:24:11.811596 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.811569 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003a5fb2-619a-4c83-be53-7a825cc9b373-kube-api-access-rvjsm" (OuterVolumeSpecName: "kube-api-access-rvjsm") pod "003a5fb2-619a-4c83-be53-7a825cc9b373" (UID: "003a5fb2-619a-4c83-be53-7a825cc9b373"). InnerVolumeSpecName "kube-api-access-rvjsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:24:11.910737 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.910687 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjsm\" (UniqueName: \"kubernetes.io/projected/003a5fb2-619a-4c83-be53-7a825cc9b373-kube-api-access-rvjsm\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:24:11.910737 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:11.910733 2585 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/003a5fb2-619a-4c83-be53-7a825cc9b373-webhook-cert\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:24:12.244409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.244373 2585 generic.go:356] "Generic (PLEG): container finished" podID="003a5fb2-619a-4c83-be53-7a825cc9b373" containerID="2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581" exitCode=0 Jul 24 00:24:12.244593 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.244422 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" Jul 24 00:24:12.244593 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.244463 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" event={"ID":"003a5fb2-619a-4c83-be53-7a825cc9b373","Type":"ContainerDied","Data":"2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581"} Jul 24 00:24:12.244593 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.244516 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bbd68d9-9mr4z" event={"ID":"003a5fb2-619a-4c83-be53-7a825cc9b373","Type":"ContainerDied","Data":"c9c60455378d8a96893314662e10a726622c746c3f313c6262afb80876d64517"} Jul 24 00:24:12.244593 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.244540 2585 scope.go:117] "RemoveContainer" containerID="2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581" Jul 24 00:24:12.254852 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.254831 2585 scope.go:117] "RemoveContainer" containerID="2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581" Jul 24 00:24:12.255169 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:24:12.255148 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581\": container with ID starting with 2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581 not found: ID does not exist" containerID="2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581" Jul 24 00:24:12.255271 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.255176 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581"} err="failed to get container status \"2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581\": rpc error: code = NotFound desc = could not find container \"2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581\": container with ID starting with 2421941b4654f9876ddc945095fe26ff340b053a7d3eafcc9ef62e8eb07b5581 not found: ID does not exist" Jul 24 00:24:12.267683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.267661 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-9mr4z"] Jul 24 00:24:12.273511 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:12.273487 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-9mr4z"] Jul 24 00:24:13.800136 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:13.800099 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003a5fb2-619a-4c83-be53-7a825cc9b373" path="/var/lib/kubelet/pods/003a5fb2-619a-4c83-be53-7a825cc9b373/volumes" Jul 24 00:24:13.838521 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:24:13.838487 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:25:23.747010 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:23.746933 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:25:23.749217 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:23.749191 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:25:23.803836 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:23.803808 2585 scope.go:117] "RemoveContainer" containerID="b0ea752514b7018b299cc5a6c71f74556bc462d9b575f213849392b2db8226af" Jul 24 00:25:33.535848 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.535809 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-nwxlt"] Jul 24 00:25:33.536424 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.536405 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003a5fb2-619a-4c83-be53-7a825cc9b373" containerName="manager" Jul 24 00:25:33.536424 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.536425 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="003a5fb2-619a-4c83-be53-7a825cc9b373" containerName="manager" Jul 24 00:25:33.536512 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.536508 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="003a5fb2-619a-4c83-be53-7a825cc9b373" containerName="manager" Jul 24 00:25:33.540944 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.540928 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.544619 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.544595 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 00:25:33.544734 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.544599 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 00:25:33.544734 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.544674 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2njs6\"" Jul 24 00:25:33.550097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.550076 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-nwxlt"] Jul 24 00:25:33.643322 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.643284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhf4c\" (UniqueName: \"kubernetes.io/projected/d7a46795-810e-4942-9f0a-7b761cefd871-kube-api-access-fhf4c\") pod \"maas-controller-5b9bbd68d9-nwxlt\" (UID: \"d7a46795-810e-4942-9f0a-7b761cefd871\") " pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.643500 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.643380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a46795-810e-4942-9f0a-7b761cefd871-webhook-cert\") pod \"maas-controller-5b9bbd68d9-nwxlt\" (UID: \"d7a46795-810e-4942-9f0a-7b761cefd871\") " pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.744681 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.744646 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a46795-810e-4942-9f0a-7b761cefd871-webhook-cert\") pod \"maas-controller-5b9bbd68d9-nwxlt\" (UID: \"d7a46795-810e-4942-9f0a-7b761cefd871\") " pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.744853 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.744709 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhf4c\" (UniqueName: \"kubernetes.io/projected/d7a46795-810e-4942-9f0a-7b761cefd871-kube-api-access-fhf4c\") pod \"maas-controller-5b9bbd68d9-nwxlt\" (UID: \"d7a46795-810e-4942-9f0a-7b761cefd871\") " pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.747094 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.747075 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7a46795-810e-4942-9f0a-7b761cefd871-webhook-cert\") pod \"maas-controller-5b9bbd68d9-nwxlt\" (UID: \"d7a46795-810e-4942-9f0a-7b761cefd871\") " pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.752636 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.752616 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhf4c\" (UniqueName: \"kubernetes.io/projected/d7a46795-810e-4942-9f0a-7b761cefd871-kube-api-access-fhf4c\") pod \"maas-controller-5b9bbd68d9-nwxlt\" (UID: \"d7a46795-810e-4942-9f0a-7b761cefd871\") " pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.852385 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.852349 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:33.987195 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:33.987052 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9bbd68d9-nwxlt"] Jul 24 00:25:33.989658 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:25:33.989630 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a46795_810e_4942_9f0a_7b761cefd871.slice/crio-fc559b8fed8582a796ac1a54007f9795630caf131aad97fd2b1d417ed4633030 WatchSource:0}: Error finding container fc559b8fed8582a796ac1a54007f9795630caf131aad97fd2b1d417ed4633030: Status 404 returned error can't find the container with id fc559b8fed8582a796ac1a54007f9795630caf131aad97fd2b1d417ed4633030 Jul 24 00:25:34.721194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:34.721108 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" event={"ID":"d7a46795-810e-4942-9f0a-7b761cefd871","Type":"ContainerStarted","Data":"b05d513bac201b4c4ab54149d851317444358a18726c155f3d81a2648784ef0f"} Jul 24 00:25:34.721194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:34.721163 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" event={"ID":"d7a46795-810e-4942-9f0a-7b761cefd871","Type":"ContainerStarted","Data":"fc559b8fed8582a796ac1a54007f9795630caf131aad97fd2b1d417ed4633030"} Jul 24 00:25:34.721605 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:34.721238 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:34.737937 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:34.737894 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" podStartSLOduration=1.388139637 podStartE2EDuration="1.737877721s" totalImagesPullingTime="349.738084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:25:34.736438305 +0000 UTC m=+911.469073331" watchObservedRunningTime="2026-07-24 00:25:34.737877721 +0000 UTC m=+911.470512737" Jul 24 00:25:45.729100 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:45.729065 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9bbd68d9-nwxlt" Jul 24 00:25:59.733286 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:25:59.733247 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:26:32.733425 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:26:32.733390 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:26:42.623530 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:26:42.623487 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:26:53.031728 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:26:53.031648 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:26:58.730201 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:26:58.730163 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:27:09.426709 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:27:09.426672 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:27:19.329104 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:27:19.329073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:27:30.426338 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:27:30.426299 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:27:40.429591 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:27:40.429554 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:28:02.633196 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:28:02.633162 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:28:15.830101 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:28:15.830019 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:28:38.323125 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:28:38.323090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:28:48.727351 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:28:48.727316 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:29:03.131060 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:29:03.131023 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:29:14.325314 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:29:14.325282 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:29:23.934459 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:29:23.934419 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:29:33.329603 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:29:33.329569 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:29:43.133595 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:29:43.133551 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:30:09.507219 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:09.507177 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm"] Jul 24 00:30:09.507822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:09.507464 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" podUID="edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" containerName="manager" containerID="cri-o://07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6" gracePeriod=10 Jul 24 00:30:12.270119 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.270089 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:30:12.314214 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.314175 2585 generic.go:356] "Generic (PLEG): container finished" podID="edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" containerID="07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6" exitCode=0 Jul 24 00:30:12.314401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.314220 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" Jul 24 00:30:12.314401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.314258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" event={"ID":"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab","Type":"ContainerDied","Data":"07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6"} Jul 24 00:30:12.314401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.314339 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm" event={"ID":"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab","Type":"ContainerDied","Data":"69af87ae48e2225c33e34972b7282b68b84371245dae0511a9a67e16a8312224"} Jul 24 00:30:12.314401 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.314366 2585 scope.go:117] "RemoveContainer" containerID="07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6" Jul 24 00:30:12.326845 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.326827 2585 scope.go:117] "RemoveContainer" containerID="07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6" Jul 24 00:30:12.327166 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:30:12.327133 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6\": container with ID starting with 07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6 not found: ID does not exist" containerID="07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6" Jul 24 00:30:12.327235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.327176 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6"} err="failed to get container status \"07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6\": rpc error: code = NotFound desc = could not find container \"07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6\": container with ID starting with 07eeca98aebd519b1eb77e7595ed7cf687fac605ccdb5c5ce7c62f37c3b8ebe6 not found: ID does not exist" Jul 24 00:30:12.344639 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.344610 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-extensions-socket-volume\") pod \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " Jul 24 00:30:12.344768 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.344674 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42z59\" (UniqueName: \"kubernetes.io/projected/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-kube-api-access-42z59\") pod \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\" (UID: \"edf58ce2-c8f1-4ab2-ab73-efe5fac3faab\") " Jul 24 00:30:12.344980 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.344958 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" (UID: "edf58ce2-c8f1-4ab2-ab73-efe5fac3faab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 00:30:12.346612 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.346590 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-kube-api-access-42z59" (OuterVolumeSpecName: "kube-api-access-42z59") pod "edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" (UID: "edf58ce2-c8f1-4ab2-ab73-efe5fac3faab"). InnerVolumeSpecName "kube-api-access-42z59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:30:12.446108 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.446068 2585 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-extensions-socket-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:30:12.446108 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.446097 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42z59\" (UniqueName: \"kubernetes.io/projected/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab-kube-api-access-42z59\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:30:12.636910 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.636873 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm"] Jul 24 00:30:12.640383 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:12.640357 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bfwpm"] Jul 24 00:30:13.799550 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:13.799519 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" path="/var/lib/kubelet/pods/edf58ce2-c8f1-4ab2-ab73-efe5fac3faab/volumes" Jul 24 00:30:23.782599 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:23.782572 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:30:23.785784 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:30:23.785757 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:31:10.743291 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.743255 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn"] Jul 24 00:31:10.745780 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.743697 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" containerName="manager" Jul 24 00:31:10.745780 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.743716 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" containerName="manager" Jul 24 00:31:10.745780 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.743918 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf58ce2-c8f1-4ab2-ab73-efe5fac3faab" containerName="manager" Jul 24 00:31:10.746896 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.746881 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:10.751261 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.751241 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-48nbf\"" Jul 24 00:31:10.758355 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.758336 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn"] Jul 24 00:31:10.874832 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.874793 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f3a182-f0ff-4c0f-bf17-2e8e458f7808-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-95rwn\" (UID: \"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:10.875029 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.874852 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788mb\" (UniqueName: \"kubernetes.io/projected/e3f3a182-f0ff-4c0f-bf17-2e8e458f7808-kube-api-access-788mb\") pod \"kuadrant-operator-controller-manager-86878c769f-95rwn\" (UID: \"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:10.976007 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.975967 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788mb\" (UniqueName: \"kubernetes.io/projected/e3f3a182-f0ff-4c0f-bf17-2e8e458f7808-kube-api-access-788mb\") pod \"kuadrant-operator-controller-manager-86878c769f-95rwn\" (UID: \"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:10.976157 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.976053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f3a182-f0ff-4c0f-bf17-2e8e458f7808-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-95rwn\" (UID: \"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:10.976387 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.976372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f3a182-f0ff-4c0f-bf17-2e8e458f7808-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-95rwn\" (UID: \"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:10.984511 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:10.984484 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788mb\" (UniqueName: \"kubernetes.io/projected/e3f3a182-f0ff-4c0f-bf17-2e8e458f7808-kube-api-access-788mb\") pod \"kuadrant-operator-controller-manager-86878c769f-95rwn\" (UID: \"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:11.057191 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:11.057165 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:11.197155 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:11.197131 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn"] Jul 24 00:31:11.198884 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:31:11.198829 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3a182_f0ff_4c0f_bf17_2e8e458f7808.slice/crio-dc754275b485eecdecd606188b0224b5e9977647d446b2b35a16260bceee2aa9 WatchSource:0}: Error finding container dc754275b485eecdecd606188b0224b5e9977647d446b2b35a16260bceee2aa9: Status 404 returned error can't find the container with id dc754275b485eecdecd606188b0224b5e9977647d446b2b35a16260bceee2aa9 Jul 24 00:31:11.641165 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:11.641129 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" event={"ID":"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808","Type":"ContainerStarted","Data":"b66bfba4ffc2b0556e6167b609d7a4997c1dc6a29dcea77880f67f40b8854028"} Jul 24 00:31:11.641350 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:11.641180 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" event={"ID":"e3f3a182-f0ff-4c0f-bf17-2e8e458f7808","Type":"ContainerStarted","Data":"dc754275b485eecdecd606188b0224b5e9977647d446b2b35a16260bceee2aa9"} Jul 24 00:31:11.641350 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:11.641201 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:11.661990 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:11.661934 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" podStartSLOduration=1.661913695 podStartE2EDuration="1.661913695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:31:11.66077133 +0000 UTC m=+1248.393406356" watchObservedRunningTime="2026-07-24 00:31:11.661913695 +0000 UTC m=+1248.394548717" Jul 24 00:31:22.649062 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:22.648988 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-95rwn" Jul 24 00:31:33.239925 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:33.239892 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:31:44.628335 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:44.628297 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:31:53.447280 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:31:53.447242 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:32:06.455394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:32:06.450777 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:32:16.227352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:32:16.227310 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:32:27.967014 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:32:27.966978 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:32:36.245639 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:32:36.245605 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:32:46.755943 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:32:46.755842 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:32:58.340479 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:32:58.340446 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:33:07.574618 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:33:07.574574 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:33:18.343115 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:33:18.343085 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:33:40.330020 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:33:40.329987 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:33:49.324575 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:33:49.324531 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:34:02.928453 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:34:02.928422 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:34:17.138200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:34:17.138122 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:34:43.443857 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:34:43.443818 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:34:56.857223 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:34:56.857186 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:35:06.833370 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:06.833338 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:35:19.437705 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:19.437665 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:35:23.823805 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:23.823779 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:35:23.837090 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:23.837068 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:35:33.238645 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:33.238601 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:35:42.137057 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:42.137021 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:35:56.341102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:35:56.341029 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:36:05.229576 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:36:05.229540 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:36:18.044084 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:36:18.044046 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:36:29.532455 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:36:29.532418 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:36:38.939297 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:36:38.939245 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:36:49.026536 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:36:49.026507 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:37:02.135745 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:37:02.135714 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:37:10.030569 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:37:10.030530 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:37:22.331105 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:37:22.331027 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:37:34.043129 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:37:34.043088 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:37:47.232222 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:37:47.232186 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:37:56.328279 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:37:56.328247 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:38:18.548063 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:18.548028 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:38:29.635484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:29.635447 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:38:51.571978 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.571895 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd"] Jul 24 00:38:51.575588 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.575570 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:51.578273 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.578249 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:38:51.584324 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.584212 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd"] Jul 24 00:38:51.596414 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.596381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082f2c86-bcac-48d9-a917-56a4202c8a53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd\" (UID: \"082f2c86-bcac-48d9-a917-56a4202c8a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:51.697331 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.697292 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082f2c86-bcac-48d9-a917-56a4202c8a53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd\" (UID: \"082f2c86-bcac-48d9-a917-56a4202c8a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:51.697925 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.697907 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082f2c86-bcac-48d9-a917-56a4202c8a53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd\" (UID: \"082f2c86-bcac-48d9-a917-56a4202c8a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:51.889414 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:51.889329 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:52.238048 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:52.236571 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd"] Jul 24 00:38:52.240836 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:38:52.240804 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082f2c86_bcac_48d9_a917_56a4202c8a53.slice/crio-bf1ccdaaf7dcbd2d77aefb6395c85615d3806c97bfdcb9a1b32388529ccf194c WatchSource:0}: Error finding container bf1ccdaaf7dcbd2d77aefb6395c85615d3806c97bfdcb9a1b32388529ccf194c: Status 404 returned error can't find the container with id bf1ccdaaf7dcbd2d77aefb6395c85615d3806c97bfdcb9a1b32388529ccf194c Jul 24 00:38:52.242676 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:52.242658 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:38:52.279625 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:52.279588 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" event={"ID":"082f2c86-bcac-48d9-a917-56a4202c8a53","Type":"ContainerStarted","Data":"bf1ccdaaf7dcbd2d77aefb6395c85615d3806c97bfdcb9a1b32388529ccf194c"} Jul 24 00:38:55.299525 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:55.299498 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" event={"ID":"082f2c86-bcac-48d9-a917-56a4202c8a53","Type":"ContainerStarted","Data":"97e68f36e00bf3ad715d1ae1d66da5befc7fce03c660961a7465a5e344bbfd11"} Jul 24 00:38:55.316938 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:55.316892 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" podStartSLOduration=1.530443964 podStartE2EDuration="4.316877696s" totalImagesPullingTime="2.786433732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:38:55.315949323 +0000 UTC m=+1712.048584376" watchObservedRunningTime="2026-07-24 00:38:55.316877696 +0000 UTC m=+1712.049512711" Jul 24 00:38:56.306269 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:56.306233 2585 generic.go:356] "Generic (PLEG): container finished" podID="082f2c86-bcac-48d9-a917-56a4202c8a53" containerID="97e68f36e00bf3ad715d1ae1d66da5befc7fce03c660961a7465a5e344bbfd11" exitCode=0 Jul 24 00:38:56.306661 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:56.306314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" event={"ID":"082f2c86-bcac-48d9-a917-56a4202c8a53","Type":"ContainerDied","Data":"97e68f36e00bf3ad715d1ae1d66da5befc7fce03c660961a7465a5e344bbfd11"} Jul 24 00:38:57.453700 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:57.453675 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:57.549971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:57.549936 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082f2c86-bcac-48d9-a917-56a4202c8a53-maas-api-service-ca\") pod \"082f2c86-bcac-48d9-a917-56a4202c8a53\" (UID: \"082f2c86-bcac-48d9-a917-56a4202c8a53\") " Jul 24 00:38:57.550314 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:57.550288 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082f2c86-bcac-48d9-a917-56a4202c8a53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "082f2c86-bcac-48d9-a917-56a4202c8a53" (UID: "082f2c86-bcac-48d9-a917-56a4202c8a53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:38:57.651091 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:57.651004 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/082f2c86-bcac-48d9-a917-56a4202c8a53-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:38:58.318619 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:58.318585 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" Jul 24 00:38:58.318794 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:58.318618 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd" event={"ID":"082f2c86-bcac-48d9-a917-56a4202c8a53","Type":"ContainerDied","Data":"bf1ccdaaf7dcbd2d77aefb6395c85615d3806c97bfdcb9a1b32388529ccf194c"} Jul 24 00:38:58.318794 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:38:58.318694 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1ccdaaf7dcbd2d77aefb6395c85615d3806c97bfdcb9a1b32388529ccf194c" Jul 24 00:39:04.927502 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:04.927462 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:39:06.661853 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:06.661818 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd"] Jul 24 00:39:06.664102 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:06.664082 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-42e5f403-dvmtd"] Jul 24 00:39:07.800161 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:07.800124 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082f2c86-bcac-48d9-a917-56a4202c8a53" path="/var/lib/kubelet/pods/082f2c86-bcac-48d9-a917-56a4202c8a53/volumes" Jul 24 00:39:11.587812 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.587774 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn"] Jul 24 00:39:11.588565 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.588540 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082f2c86-bcac-48d9-a917-56a4202c8a53" containerName="revoke-keys" Jul 24 00:39:11.588565 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.588566 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="082f2c86-bcac-48d9-a917-56a4202c8a53" containerName="revoke-keys" Jul 24 00:39:11.588726 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.588673 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="082f2c86-bcac-48d9-a917-56a4202c8a53" containerName="revoke-keys" Jul 24 00:39:11.592297 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.592266 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.595038 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.595012 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8405249f\"" Jul 24 00:39:11.595166 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.595076 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8405249f-dockercfg-2hs88\"" Jul 24 00:39:11.601975 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.601944 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn"] Jul 24 00:39:11.680346 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.680308 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8wk\" (UniqueName: \"kubernetes.io/projected/522b7a3b-857e-451f-a703-09a85091b5b3-kube-api-access-ct8wk\") pod \"payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.680517 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.680435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b7a3b-857e-451f-a703-09a85091b5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.781875 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.781820 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b7a3b-857e-451f-a703-09a85091b5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.782044 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.781973 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8wk\" (UniqueName: \"kubernetes.io/projected/522b7a3b-857e-451f-a703-09a85091b5b3-kube-api-access-ct8wk\") pod \"payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.782559 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.782527 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b7a3b-857e-451f-a703-09a85091b5b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.790264 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.790237 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8wk\" (UniqueName: \"kubernetes.io/projected/522b7a3b-857e-451f-a703-09a85091b5b3-kube-api-access-ct8wk\") pod \"payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:11.905123 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:11.905036 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:12.258022 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:12.257962 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn"] Jul 24 00:39:12.261229 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:39:12.261203 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522b7a3b_857e_451f_a703_09a85091b5b3.slice/crio-94929a5c10468caf315f4a1b1d150402204a86f67f8267650ac0758bbade60fe WatchSource:0}: Error finding container 94929a5c10468caf315f4a1b1d150402204a86f67f8267650ac0758bbade60fe: Status 404 returned error can't find the container with id 94929a5c10468caf315f4a1b1d150402204a86f67f8267650ac0758bbade60fe Jul 24 00:39:12.402972 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:12.402938 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" event={"ID":"522b7a3b-857e-451f-a703-09a85091b5b3","Type":"ContainerStarted","Data":"f82a51d1c720b1a3b0056b10e08566ce3db76e69a30d129ada0cb7b67a95cf5d"} Jul 24 00:39:12.403125 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:12.402996 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" event={"ID":"522b7a3b-857e-451f-a703-09a85091b5b3","Type":"ContainerStarted","Data":"94929a5c10468caf315f4a1b1d150402204a86f67f8267650ac0758bbade60fe"} Jul 24 00:39:12.403125 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:12.403024 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:12.435920 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:12.435847 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" podStartSLOduration=1.435831141 podStartE2EDuration="1.435831141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:39:12.432674683 +0000 UTC m=+1729.165309708" watchObservedRunningTime="2026-07-24 00:39:12.435831141 +0000 UTC m=+1729.168466166" Jul 24 00:39:18.151841 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:18.151806 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:39:23.411288 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:23.411259 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:46.422561 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.422522 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn"] Jul 24 00:39:46.423043 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.422774 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" podUID="522b7a3b-857e-451f-a703-09a85091b5b3" containerName="payload-pre-processing" containerID="cri-o://f82a51d1c720b1a3b0056b10e08566ce3db76e69a30d129ada0cb7b67a95cf5d" gracePeriod=30 Jul 24 00:39:46.612498 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.612469 2585 generic.go:356] "Generic (PLEG): container finished" podID="522b7a3b-857e-451f-a703-09a85091b5b3" containerID="f82a51d1c720b1a3b0056b10e08566ce3db76e69a30d129ada0cb7b67a95cf5d" exitCode=0 Jul 24 00:39:46.612644 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.612543 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" event={"ID":"522b7a3b-857e-451f-a703-09a85091b5b3","Type":"ContainerDied","Data":"f82a51d1c720b1a3b0056b10e08566ce3db76e69a30d129ada0cb7b67a95cf5d"} Jul 24 00:39:46.694565 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.694538 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:46.795877 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.795842 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct8wk\" (UniqueName: \"kubernetes.io/projected/522b7a3b-857e-451f-a703-09a85091b5b3-kube-api-access-ct8wk\") pod \"522b7a3b-857e-451f-a703-09a85091b5b3\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " Jul 24 00:39:46.796076 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.795957 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b7a3b-857e-451f-a703-09a85091b5b3-plugins-config-volume\") pod \"522b7a3b-857e-451f-a703-09a85091b5b3\" (UID: \"522b7a3b-857e-451f-a703-09a85091b5b3\") " Jul 24 00:39:46.796355 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.796332 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522b7a3b-857e-451f-a703-09a85091b5b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "522b7a3b-857e-451f-a703-09a85091b5b3" (UID: "522b7a3b-857e-451f-a703-09a85091b5b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:39:46.797955 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.797932 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522b7a3b-857e-451f-a703-09a85091b5b3-kube-api-access-ct8wk" (OuterVolumeSpecName: "kube-api-access-ct8wk") pod "522b7a3b-857e-451f-a703-09a85091b5b3" (UID: "522b7a3b-857e-451f-a703-09a85091b5b3"). InnerVolumeSpecName "kube-api-access-ct8wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:39:46.897061 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.897025 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct8wk\" (UniqueName: \"kubernetes.io/projected/522b7a3b-857e-451f-a703-09a85091b5b3-kube-api-access-ct8wk\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:39:46.897061 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:46.897054 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522b7a3b-857e-451f-a703-09a85091b5b3-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:39:47.619840 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:47.619802 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" event={"ID":"522b7a3b-857e-451f-a703-09a85091b5b3","Type":"ContainerDied","Data":"94929a5c10468caf315f4a1b1d150402204a86f67f8267650ac0758bbade60fe"} Jul 24 00:39:47.619840 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:47.619819 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn" Jul 24 00:39:47.620379 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:47.619898 2585 scope.go:117] "RemoveContainer" containerID="f82a51d1c720b1a3b0056b10e08566ce3db76e69a30d129ada0cb7b67a95cf5d" Jul 24 00:39:47.643475 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:47.643451 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn"] Jul 24 00:39:47.647741 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:47.647719 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8405249f-5978cf557b-x8rrn"] Jul 24 00:39:47.800282 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:47.800254 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522b7a3b-857e-451f-a703-09a85091b5b3" path="/var/lib/kubelet/pods/522b7a3b-857e-451f-a703-09a85091b5b3/volumes" Jul 24 00:39:49.330481 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:39:49.330447 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:40:01.936452 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:01.936420 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:40:23.863059 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:23.863032 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:40:23.876842 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:23.876813 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:40:33.835359 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:33.835324 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:40:40.888855 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.888805 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk"] Jul 24 00:40:40.889362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.889344 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522b7a3b-857e-451f-a703-09a85091b5b3" containerName="payload-pre-processing" Jul 24 00:40:40.889408 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.889364 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="522b7a3b-857e-451f-a703-09a85091b5b3" containerName="payload-pre-processing" Jul 24 00:40:40.889444 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.889436 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="522b7a3b-857e-451f-a703-09a85091b5b3" containerName="payload-pre-processing" Jul 24 00:40:40.892593 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.892578 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:40.895956 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.895923 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bd6ad2bc-dockercfg-7mzjg\"" Jul 24 00:40:40.896282 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.896259 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bd6ad2bc\"" Jul 24 00:40:40.903027 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.903002 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk"] Jul 24 00:40:40.990130 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.990094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f939cf-f757-49aa-8f82-ec534ec99800-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:40.990287 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:40.990247 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pc4\" (UniqueName: \"kubernetes.io/projected/c7f939cf-f757-49aa-8f82-ec534ec99800-kube-api-access-m5pc4\") pod \"payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.091728 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.091690 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pc4\" (UniqueName: \"kubernetes.io/projected/c7f939cf-f757-49aa-8f82-ec534ec99800-kube-api-access-m5pc4\") pod \"payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.091728 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.091748 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f939cf-f757-49aa-8f82-ec534ec99800-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.092632 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.092605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f939cf-f757-49aa-8f82-ec534ec99800-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.100592 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.100566 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pc4\" (UniqueName: \"kubernetes.io/projected/c7f939cf-f757-49aa-8f82-ec534ec99800-kube-api-access-m5pc4\") pod \"payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.204183 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.204096 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.345113 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.345073 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk"] Jul 24 00:40:41.347047 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:40:41.347023 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f939cf_f757_49aa_8f82_ec534ec99800.slice/crio-c7cf0d851827d318dcdde4f61ec68ccb8b98174d5cf101b813ce9e99b37acaa8 WatchSource:0}: Error finding container c7cf0d851827d318dcdde4f61ec68ccb8b98174d5cf101b813ce9e99b37acaa8: Status 404 returned error can't find the container with id c7cf0d851827d318dcdde4f61ec68ccb8b98174d5cf101b813ce9e99b37acaa8 Jul 24 00:40:41.926348 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.926310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" event={"ID":"c7f939cf-f757-49aa-8f82-ec534ec99800","Type":"ContainerStarted","Data":"18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c"} Jul 24 00:40:41.926803 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.926370 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:40:41.926803 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.926388 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" event={"ID":"c7f939cf-f757-49aa-8f82-ec534ec99800","Type":"ContainerStarted","Data":"c7cf0d851827d318dcdde4f61ec68ccb8b98174d5cf101b813ce9e99b37acaa8"} Jul 24 00:40:41.941981 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:41.941933 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" podStartSLOduration=1.94191816 podStartE2EDuration="1.94191816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:40:41.941211777 +0000 UTC m=+1818.673846796" watchObservedRunningTime="2026-07-24 00:40:41.94191816 +0000 UTC m=+1818.674553185" Jul 24 00:40:45.237106 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:45.237063 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:40:52.934665 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:40:52.934632 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:41:15.545696 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.545662 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk"] Jul 24 00:41:15.546315 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.546250 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" podUID="c7f939cf-f757-49aa-8f82-ec534ec99800" containerName="payload-pre-processing" containerID="cri-o://18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c" gracePeriod=30 Jul 24 00:41:15.837631 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.837602 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:41:15.988458 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.988425 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f939cf-f757-49aa-8f82-ec534ec99800-plugins-config-volume\") pod \"c7f939cf-f757-49aa-8f82-ec534ec99800\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " Jul 24 00:41:15.988635 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.988521 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5pc4\" (UniqueName: \"kubernetes.io/projected/c7f939cf-f757-49aa-8f82-ec534ec99800-kube-api-access-m5pc4\") pod \"c7f939cf-f757-49aa-8f82-ec534ec99800\" (UID: \"c7f939cf-f757-49aa-8f82-ec534ec99800\") " Jul 24 00:41:15.988806 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.988783 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f939cf-f757-49aa-8f82-ec534ec99800-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7f939cf-f757-49aa-8f82-ec534ec99800" (UID: "c7f939cf-f757-49aa-8f82-ec534ec99800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:41:15.990577 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:15.990557 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f939cf-f757-49aa-8f82-ec534ec99800-kube-api-access-m5pc4" (OuterVolumeSpecName: "kube-api-access-m5pc4") pod "c7f939cf-f757-49aa-8f82-ec534ec99800" (UID: "c7f939cf-f757-49aa-8f82-ec534ec99800"). InnerVolumeSpecName "kube-api-access-m5pc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:41:16.089191 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.089106 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pc4\" (UniqueName: \"kubernetes.io/projected/c7f939cf-f757-49aa-8f82-ec534ec99800-kube-api-access-m5pc4\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:41:16.089191 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.089136 2585 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f939cf-f757-49aa-8f82-ec534ec99800-plugins-config-volume\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:41:16.120362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.120335 2585 generic.go:356] "Generic (PLEG): container finished" podID="c7f939cf-f757-49aa-8f82-ec534ec99800" containerID="18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c" exitCode=0 Jul 24 00:41:16.120513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.120378 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" Jul 24 00:41:16.120513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.120412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" event={"ID":"c7f939cf-f757-49aa-8f82-ec534ec99800","Type":"ContainerDied","Data":"18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c"} Jul 24 00:41:16.120513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.120451 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk" event={"ID":"c7f939cf-f757-49aa-8f82-ec534ec99800","Type":"ContainerDied","Data":"c7cf0d851827d318dcdde4f61ec68ccb8b98174d5cf101b813ce9e99b37acaa8"} Jul 24 00:41:16.120513 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.120473 2585 scope.go:117] "RemoveContainer" containerID="18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c" Jul 24 00:41:16.129673 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.129654 2585 scope.go:117] "RemoveContainer" containerID="18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c" Jul 24 00:41:16.129933 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:41:16.129915 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c\": container with ID starting with 18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c not found: ID does not exist" containerID="18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c" Jul 24 00:41:16.129990 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.129941 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c"} err="failed to get container status \"18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c\": rpc error: code = NotFound desc = could not find container \"18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c\": container with ID starting with 18dc5643c0406da4d5d381ced567af6a987411b63c424a828f4dcd6db3f66f8c not found: ID does not exist" Jul 24 00:41:16.141615 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.141593 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk"] Jul 24 00:41:16.145239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:16.145219 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bd6ad2bc-778f955888-wp2xk"] Jul 24 00:41:17.800148 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:17.800112 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f939cf-f757-49aa-8f82-ec534ec99800" path="/var/lib/kubelet/pods/c7f939cf-f757-49aa-8f82-ec534ec99800/volumes" Jul 24 00:41:18.635180 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:41:18.635146 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:42:46.328701 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:42:46.328663 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:42:56.731296 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:42:56.731257 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:43:06.135719 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:06.135680 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:43:12.535037 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.535005 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm"] Jul 24 00:43:12.535479 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.535463 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f939cf-f757-49aa-8f82-ec534ec99800" containerName="payload-pre-processing" Jul 24 00:43:12.535554 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.535480 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f939cf-f757-49aa-8f82-ec534ec99800" containerName="payload-pre-processing" Jul 24 00:43:12.535606 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.535563 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f939cf-f757-49aa-8f82-ec534ec99800" containerName="payload-pre-processing" Jul 24 00:43:12.538815 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.538797 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:12.541450 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.541431 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:43:12.546395 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.546369 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm"] Jul 24 00:43:12.626104 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.626066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb63855-e3be-4570-a9fb-118b418117e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm\" (UID: \"4eb63855-e3be-4570-a9fb-118b418117e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:12.727074 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.727044 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb63855-e3be-4570-a9fb-118b418117e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm\" (UID: \"4eb63855-e3be-4570-a9fb-118b418117e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:12.727698 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.727678 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb63855-e3be-4570-a9fb-118b418117e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm\" (UID: \"4eb63855-e3be-4570-a9fb-118b418117e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:12.852423 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.852390 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:12.987844 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:12.987814 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm"] Jul 24 00:43:12.988477 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:43:12.988449 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb63855_e3be_4570_a9fb_118b418117e3.slice/crio-64470973cef24fc823356ce990b9d6622cead2f2af84f5f428841ab87fe2e857 WatchSource:0}: Error finding container 64470973cef24fc823356ce990b9d6622cead2f2af84f5f428841ab87fe2e857: Status 404 returned error can't find the container with id 64470973cef24fc823356ce990b9d6622cead2f2af84f5f428841ab87fe2e857 Jul 24 00:43:13.791927 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:13.791896 2585 generic.go:356] "Generic (PLEG): container finished" podID="4eb63855-e3be-4570-a9fb-118b418117e3" containerID="f4edaa44f4e3e3a6a514449e271a49d59e7fafdf945bb8ee64c0d50a3e756d7e" exitCode=0 Jul 24 00:43:13.792259 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:13.791986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" event={"ID":"4eb63855-e3be-4570-a9fb-118b418117e3","Type":"ContainerDied","Data":"f4edaa44f4e3e3a6a514449e271a49d59e7fafdf945bb8ee64c0d50a3e756d7e"} Jul 24 00:43:13.792259 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:13.792039 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" event={"ID":"4eb63855-e3be-4570-a9fb-118b418117e3","Type":"ContainerStarted","Data":"64470973cef24fc823356ce990b9d6622cead2f2af84f5f428841ab87fe2e857"} Jul 24 00:43:14.931235 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:14.931204 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:15.052159 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:15.052079 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb63855-e3be-4570-a9fb-118b418117e3-maas-api-service-ca\") pod \"4eb63855-e3be-4570-a9fb-118b418117e3\" (UID: \"4eb63855-e3be-4570-a9fb-118b418117e3\") " Jul 24 00:43:15.052434 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:15.052412 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb63855-e3be-4570-a9fb-118b418117e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4eb63855-e3be-4570-a9fb-118b418117e3" (UID: "4eb63855-e3be-4570-a9fb-118b418117e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:43:15.152963 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:15.152930 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb63855-e3be-4570-a9fb-118b418117e3-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:43:15.804625 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:15.804453 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" Jul 24 00:43:15.804625 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:15.804483 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm" event={"ID":"4eb63855-e3be-4570-a9fb-118b418117e3","Type":"ContainerDied","Data":"64470973cef24fc823356ce990b9d6622cead2f2af84f5f428841ab87fe2e857"} Jul 24 00:43:15.804625 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:15.804529 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64470973cef24fc823356ce990b9d6622cead2f2af84f5f428841ab87fe2e857" Jul 24 00:43:26.130630 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:26.130591 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:43:27.608576 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:27.608538 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm"] Jul 24 00:43:27.610894 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:27.610874 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f92c6ce-jfsrm"] Jul 24 00:43:27.800239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:43:27.800193 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb63855-e3be-4570-a9fb-118b418117e3" path="/var/lib/kubelet/pods/4eb63855-e3be-4570-a9fb-118b418117e3/volumes" Jul 24 00:44:03.139357 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:03.139320 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:44:14.335619 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:14.335586 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:44:26.635143 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:26.635108 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:44:30.553074 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.553030 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg"] Jul 24 00:44:30.553538 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.553485 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb63855-e3be-4570-a9fb-118b418117e3" containerName="revoke-keys" Jul 24 00:44:30.553538 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.553497 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb63855-e3be-4570-a9fb-118b418117e3" containerName="revoke-keys" Jul 24 00:44:30.553655 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.553573 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb63855-e3be-4570-a9fb-118b418117e3" containerName="revoke-keys" Jul 24 00:44:30.556686 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.556666 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:30.559130 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.559112 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:44:30.564275 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.564123 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg"] Jul 24 00:44:30.751636 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.751599 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8611a027-0cd4-4910-8945-ed7f950a1bd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464c3345-2pxjg\" (UID: \"8611a027-0cd4-4910-8945-ed7f950a1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:30.852138 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.852102 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8611a027-0cd4-4910-8945-ed7f950a1bd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464c3345-2pxjg\" (UID: \"8611a027-0cd4-4910-8945-ed7f950a1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:30.852731 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.852710 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8611a027-0cd4-4910-8945-ed7f950a1bd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-464c3345-2pxjg\" (UID: \"8611a027-0cd4-4910-8945-ed7f950a1bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:30.868020 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:30.867995 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:31.009959 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:31.009931 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg"] Jul 24 00:44:31.011414 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:44:31.011388 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8611a027_0cd4_4910_8945_ed7f950a1bd9.slice/crio-55d8cfc5cb8300cbe7f13a59861d2070617fbe6851c986bc9343d0398d05a9a4 WatchSource:0}: Error finding container 55d8cfc5cb8300cbe7f13a59861d2070617fbe6851c986bc9343d0398d05a9a4: Status 404 returned error can't find the container with id 55d8cfc5cb8300cbe7f13a59861d2070617fbe6851c986bc9343d0398d05a9a4 Jul 24 00:44:31.230798 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:31.230760 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" event={"ID":"8611a027-0cd4-4910-8945-ed7f950a1bd9","Type":"ContainerStarted","Data":"55d8cfc5cb8300cbe7f13a59861d2070617fbe6851c986bc9343d0398d05a9a4"} Jul 24 00:44:32.237393 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:32.237358 2585 generic.go:356] "Generic (PLEG): container finished" podID="8611a027-0cd4-4910-8945-ed7f950a1bd9" containerID="78d44d4e661d7cc62c257798d12cf50d6de38ac1643945b6f8d2667357398eab" exitCode=0 Jul 24 00:44:32.237764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:32.237445 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" event={"ID":"8611a027-0cd4-4910-8945-ed7f950a1bd9","Type":"ContainerDied","Data":"78d44d4e661d7cc62c257798d12cf50d6de38ac1643945b6f8d2667357398eab"} Jul 24 00:44:33.369139 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:33.369110 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:33.385674 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:33.385340 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8611a027-0cd4-4910-8945-ed7f950a1bd9-maas-api-service-ca\") pod \"8611a027-0cd4-4910-8945-ed7f950a1bd9\" (UID: \"8611a027-0cd4-4910-8945-ed7f950a1bd9\") " Jul 24 00:44:33.386223 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:33.386193 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8611a027-0cd4-4910-8945-ed7f950a1bd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8611a027-0cd4-4910-8945-ed7f950a1bd9" (UID: "8611a027-0cd4-4910-8945-ed7f950a1bd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:44:33.487184 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:33.487147 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8611a027-0cd4-4910-8945-ed7f950a1bd9-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:44:34.249365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:34.249288 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" event={"ID":"8611a027-0cd4-4910-8945-ed7f950a1bd9","Type":"ContainerDied","Data":"55d8cfc5cb8300cbe7f13a59861d2070617fbe6851c986bc9343d0398d05a9a4"} Jul 24 00:44:34.249365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:34.249314 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg" Jul 24 00:44:34.249365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:34.249337 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d8cfc5cb8300cbe7f13a59861d2070617fbe6851c986bc9343d0398d05a9a4" Jul 24 00:44:44.726333 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:44.726297 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:44:50.630781 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:50.630700 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg"] Jul 24 00:44:50.632971 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:50.632949 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-464c3345-2pxjg"] Jul 24 00:44:51.799110 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:51.799076 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8611a027-0cd4-4910-8945-ed7f950a1bd9" path="/var/lib/kubelet/pods/8611a027-0cd4-4910-8945-ed7f950a1bd9/volumes" Jul 24 00:44:54.648339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:44:54.648307 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:45:23.899483 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:45:23.899455 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:45:23.916642 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:45:23.916618 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:45:24.006365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:45:24.006334 2585 scope.go:117] "RemoveContainer" containerID="97e68f36e00bf3ad715d1ae1d66da5befc7fce03c660961a7465a5e344bbfd11" Jul 24 00:45:45.944225 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:45:45.944191 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:45:55.530590 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:45:55.530554 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:46:46.547170 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:46:46.547134 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:46:58.829694 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:46:58.829656 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:47:08.534396 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:47:08.534362 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:47:32.331624 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:47:32.331590 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:47:46.330841 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:47:46.330762 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:48:09.278635 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.278601 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t"] Jul 24 00:48:09.279200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.279020 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8611a027-0cd4-4910-8945-ed7f950a1bd9" containerName="revoke-keys" Jul 24 00:48:09.279200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.279032 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8611a027-0cd4-4910-8945-ed7f950a1bd9" containerName="revoke-keys" Jul 24 00:48:09.279200 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.279108 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8611a027-0cd4-4910-8945-ed7f950a1bd9" containerName="revoke-keys" Jul 24 00:48:09.282031 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.282008 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:09.284628 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.284605 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:48:09.288973 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.288953 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t"] Jul 24 00:48:09.383954 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.383924 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24087a3-008a-49e5-a5e3-285c69ead0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c28612-8k69t\" (UID: \"b24087a3-008a-49e5-a5e3-285c69ead0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:09.485015 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.484979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24087a3-008a-49e5-a5e3-285c69ead0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c28612-8k69t\" (UID: \"b24087a3-008a-49e5-a5e3-285c69ead0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:09.485633 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.485614 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24087a3-008a-49e5-a5e3-285c69ead0d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c28612-8k69t\" (UID: \"b24087a3-008a-49e5-a5e3-285c69ead0d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:09.593942 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.593917 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:09.728129 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:09.728103 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t"] Jul 24 00:48:09.729691 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:48:09.729653 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24087a3_008a_49e5_a5e3_285c69ead0d7.slice/crio-e4ea5791a1c015cd7c15f3892a1c96cfe7141c11a8f7f73318261b4a4e5075e2 WatchSource:0}: Error finding container e4ea5791a1c015cd7c15f3892a1c96cfe7141c11a8f7f73318261b4a4e5075e2: Status 404 returned error can't find the container with id e4ea5791a1c015cd7c15f3892a1c96cfe7141c11a8f7f73318261b4a4e5075e2 Jul 24 00:48:10.487495 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:10.487457 2585 generic.go:356] "Generic (PLEG): container finished" podID="b24087a3-008a-49e5-a5e3-285c69ead0d7" containerID="bd24fa695fa00da7488eb865e3615a210e86c1c8b080a41d5d68af010815a8dd" exitCode=0 Jul 24 00:48:10.487874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:10.487533 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" event={"ID":"b24087a3-008a-49e5-a5e3-285c69ead0d7","Type":"ContainerDied","Data":"bd24fa695fa00da7488eb865e3615a210e86c1c8b080a41d5d68af010815a8dd"} Jul 24 00:48:10.487874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:10.487579 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" event={"ID":"b24087a3-008a-49e5-a5e3-285c69ead0d7","Type":"ContainerStarted","Data":"e4ea5791a1c015cd7c15f3892a1c96cfe7141c11a8f7f73318261b4a4e5075e2"} Jul 24 00:48:11.622346 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:11.622324 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:11.704532 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:11.704507 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24087a3-008a-49e5-a5e3-285c69ead0d7-maas-api-service-ca\") pod \"b24087a3-008a-49e5-a5e3-285c69ead0d7\" (UID: \"b24087a3-008a-49e5-a5e3-285c69ead0d7\") " Jul 24 00:48:11.704872 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:11.704835 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24087a3-008a-49e5-a5e3-285c69ead0d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b24087a3-008a-49e5-a5e3-285c69ead0d7" (UID: "b24087a3-008a-49e5-a5e3-285c69ead0d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:48:11.805168 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:11.805146 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24087a3-008a-49e5-a5e3-285c69ead0d7-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:48:12.499991 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:12.499964 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" Jul 24 00:48:12.500170 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:12.500017 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t" event={"ID":"b24087a3-008a-49e5-a5e3-285c69ead0d7","Type":"ContainerDied","Data":"e4ea5791a1c015cd7c15f3892a1c96cfe7141c11a8f7f73318261b4a4e5075e2"} Jul 24 00:48:12.500170 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:12.500068 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ea5791a1c015cd7c15f3892a1c96cfe7141c11a8f7f73318261b4a4e5075e2" Jul 24 00:48:12.662740 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:12.662710 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:48:24.546278 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:24.546243 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t"] Jul 24 00:48:24.559697 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:24.559664 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c28612-8k69t"] Jul 24 00:48:24.840572 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:24.840543 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:48:25.800497 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:25.800469 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24087a3-008a-49e5-a5e3-285c69ead0d7" path="/var/lib/kubelet/pods/b24087a3-008a-49e5-a5e3-285c69ead0d7/volumes" Jul 24 00:48:31.476421 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.476377 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d"] Jul 24 00:48:31.477059 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.477041 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24087a3-008a-49e5-a5e3-285c69ead0d7" containerName="revoke-keys" Jul 24 00:48:31.477111 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.477063 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24087a3-008a-49e5-a5e3-285c69ead0d7" containerName="revoke-keys" Jul 24 00:48:31.477170 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.477159 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24087a3-008a-49e5-a5e3-285c69ead0d7" containerName="revoke-keys" Jul 24 00:48:31.480741 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.480723 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:31.483587 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.483560 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:48:31.485967 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.485942 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d"] Jul 24 00:48:31.580376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.580339 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d\" (UID: \"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:31.681906 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.681873 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d\" (UID: \"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:31.682511 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.682491 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d\" (UID: \"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:31.794044 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.793961 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:31.936610 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:31.936583 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d"] Jul 24 00:48:31.938207 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:48:31.938183 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83c113a_f4cb_48a7_9b86_cb9acdf5f1fd.slice/crio-8117e05bd409d919268d6ce8a8f301b74adcdd3ab15842107803ac0320a1f0ee WatchSource:0}: Error finding container 8117e05bd409d919268d6ce8a8f301b74adcdd3ab15842107803ac0320a1f0ee: Status 404 returned error can't find the container with id 8117e05bd409d919268d6ce8a8f301b74adcdd3ab15842107803ac0320a1f0ee Jul 24 00:48:32.609968 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:32.609931 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" event={"ID":"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd","Type":"ContainerStarted","Data":"3fa7f281b524c35593bba2960994eb3167b3d5ff5caaccba8b0900c872f4a5c8"} Jul 24 00:48:32.610434 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:32.609981 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" event={"ID":"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd","Type":"ContainerStarted","Data":"8117e05bd409d919268d6ce8a8f301b74adcdd3ab15842107803ac0320a1f0ee"} Jul 24 00:48:32.628137 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:32.628092 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" podStartSLOduration=1.6280747930000001 podStartE2EDuration="1.628074793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:48:32.627729581 +0000 UTC m=+2289.360364605" watchObservedRunningTime="2026-07-24 00:48:32.628074793 +0000 UTC m=+2289.360709816" Jul 24 00:48:33.616802 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:33.616762 2585 generic.go:356] "Generic (PLEG): container finished" podID="e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd" containerID="3fa7f281b524c35593bba2960994eb3167b3d5ff5caaccba8b0900c872f4a5c8" exitCode=0 Jul 24 00:48:33.617192 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:33.616851 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" event={"ID":"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd","Type":"ContainerDied","Data":"3fa7f281b524c35593bba2960994eb3167b3d5ff5caaccba8b0900c872f4a5c8"} Jul 24 00:48:34.756851 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:34.756830 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:34.911974 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:34.911883 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd-maas-api-service-ca\") pod \"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd\" (UID: \"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd\") " Jul 24 00:48:34.912279 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:34.912255 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd" (UID: "e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:48:35.013210 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:35.013179 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:48:35.628658 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:35.628632 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" Jul 24 00:48:35.628836 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:35.628669 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d" event={"ID":"e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd","Type":"ContainerDied","Data":"8117e05bd409d919268d6ce8a8f301b74adcdd3ab15842107803ac0320a1f0ee"} Jul 24 00:48:35.628836 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:35.628715 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8117e05bd409d919268d6ce8a8f301b74adcdd3ab15842107803ac0320a1f0ee" Jul 24 00:48:45.148669 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:45.148633 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:48:46.548204 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:46.548170 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d"] Jul 24 00:48:46.551527 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:46.551502 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5f040e-9wh8d"] Jul 24 00:48:47.799800 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:47.799769 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd" path="/var/lib/kubelet/pods/e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd/volumes" Jul 24 00:48:59.630414 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:48:59.630377 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:49:13.129640 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:13.129606 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:49:17.782352 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.782260 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr"] Jul 24 00:49:17.782852 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.782834 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd" containerName="revoke-keys" Jul 24 00:49:17.782852 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.782854 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd" containerName="revoke-keys" Jul 24 00:49:17.782996 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.782981 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83c113a-f4cb-48a7-9b86-cb9acdf5f1fd" containerName="revoke-keys" Jul 24 00:49:17.786365 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.786342 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:17.789683 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.789665 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:49:17.802908 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.802889 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr"] Jul 24 00:49:17.887305 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.887275 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c291daf1-eee1-49fb-abc1-521b1e948efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr\" (UID: \"c291daf1-eee1-49fb-abc1-521b1e948efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:17.988283 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.988251 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c291daf1-eee1-49fb-abc1-521b1e948efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr\" (UID: \"c291daf1-eee1-49fb-abc1-521b1e948efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:17.988877 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:17.988840 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c291daf1-eee1-49fb-abc1-521b1e948efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr\" (UID: \"c291daf1-eee1-49fb-abc1-521b1e948efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:18.101509 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:18.101480 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:18.237299 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:18.237269 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr"] Jul 24 00:49:18.238603 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:49:18.238581 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc291daf1_eee1_49fb_abc1_521b1e948efc.slice/crio-f48808feb676993da14999d0dca61e02cf5c8303a944b0498b94a2d239b37b10 WatchSource:0}: Error finding container f48808feb676993da14999d0dca61e02cf5c8303a944b0498b94a2d239b37b10: Status 404 returned error can't find the container with id f48808feb676993da14999d0dca61e02cf5c8303a944b0498b94a2d239b37b10 Jul 24 00:49:18.875305 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:18.875264 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" event={"ID":"c291daf1-eee1-49fb-abc1-521b1e948efc","Type":"ContainerStarted","Data":"037caf0cc673d1ad7fa9d34b7afb6542f1ca868c829d7fae0fe7c4ea46e995e1"} Jul 24 00:49:18.875781 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:18.875333 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" event={"ID":"c291daf1-eee1-49fb-abc1-521b1e948efc","Type":"ContainerStarted","Data":"f48808feb676993da14999d0dca61e02cf5c8303a944b0498b94a2d239b37b10"} Jul 24 00:49:18.890984 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:18.890933 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" podStartSLOduration=1.890918986 podStartE2EDuration="1.890918986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:49:18.889031179 +0000 UTC m=+2335.621666205" watchObservedRunningTime="2026-07-24 00:49:18.890918986 +0000 UTC m=+2335.623554010" Jul 24 00:49:19.883035 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:19.883002 2585 generic.go:356] "Generic (PLEG): container finished" podID="c291daf1-eee1-49fb-abc1-521b1e948efc" containerID="037caf0cc673d1ad7fa9d34b7afb6542f1ca868c829d7fae0fe7c4ea46e995e1" exitCode=0 Jul 24 00:49:19.883433 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:19.883087 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" event={"ID":"c291daf1-eee1-49fb-abc1-521b1e948efc","Type":"ContainerDied","Data":"037caf0cc673d1ad7fa9d34b7afb6542f1ca868c829d7fae0fe7c4ea46e995e1"} Jul 24 00:49:21.026470 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.026448 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:21.119822 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.119792 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c291daf1-eee1-49fb-abc1-521b1e948efc-maas-api-service-ca\") pod \"c291daf1-eee1-49fb-abc1-521b1e948efc\" (UID: \"c291daf1-eee1-49fb-abc1-521b1e948efc\") " Jul 24 00:49:21.120204 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.120156 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c291daf1-eee1-49fb-abc1-521b1e948efc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c291daf1-eee1-49fb-abc1-521b1e948efc" (UID: "c291daf1-eee1-49fb-abc1-521b1e948efc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:49:21.221217 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.221144 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c291daf1-eee1-49fb-abc1-521b1e948efc-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:49:21.895728 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.895700 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" Jul 24 00:49:21.895901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.895735 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr" event={"ID":"c291daf1-eee1-49fb-abc1-521b1e948efc","Type":"ContainerDied","Data":"f48808feb676993da14999d0dca61e02cf5c8303a944b0498b94a2d239b37b10"} Jul 24 00:49:21.895901 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:21.895784 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48808feb676993da14999d0dca61e02cf5c8303a944b0498b94a2d239b37b10" Jul 24 00:49:24.056880 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:24.056828 2585 scope.go:117] "RemoveContainer" containerID="f4edaa44f4e3e3a6a514449e271a49d59e7fafdf945bb8ee64c0d50a3e756d7e" Jul 24 00:49:37.867055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:37.867022 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr"] Jul 24 00:49:37.875578 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:37.875553 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-796b6a-pnmwr"] Jul 24 00:49:39.232808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:39.232773 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:49:39.799243 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:39.799214 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c291daf1-eee1-49fb-abc1-521b1e948efc" path="/var/lib/kubelet/pods/c291daf1-eee1-49fb-abc1-521b1e948efc/volumes" Jul 24 00:49:45.061361 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.061326 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467"] Jul 24 00:49:45.061798 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.061723 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c291daf1-eee1-49fb-abc1-521b1e948efc" containerName="revoke-keys" Jul 24 00:49:45.061798 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.061733 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c291daf1-eee1-49fb-abc1-521b1e948efc" containerName="revoke-keys" Jul 24 00:49:45.061890 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.061811 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c291daf1-eee1-49fb-abc1-521b1e948efc" containerName="revoke-keys" Jul 24 00:49:45.065092 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.065075 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:45.067775 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.067719 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:49:45.075221 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.073136 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467"] Jul 24 00:49:45.130828 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.130795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b7239e-83fc-49d8-bbf0-8fe569f31bff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a4c256-f4467\" (UID: \"b4b7239e-83fc-49d8-bbf0-8fe569f31bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:45.232233 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.232197 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b7239e-83fc-49d8-bbf0-8fe569f31bff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a4c256-f4467\" (UID: \"b4b7239e-83fc-49d8-bbf0-8fe569f31bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:45.232814 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.232797 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b7239e-83fc-49d8-bbf0-8fe569f31bff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a4c256-f4467\" (UID: \"b4b7239e-83fc-49d8-bbf0-8fe569f31bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:45.381322 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.381236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:45.520974 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:45.520948 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467"] Jul 24 00:49:45.522613 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:49:45.522587 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b7239e_83fc_49d8_bbf0_8fe569f31bff.slice/crio-cd698aa8dc7f7badc7aeeba7d71b149a7582a67dd318f6565f08c768ef494d3c WatchSource:0}: Error finding container cd698aa8dc7f7badc7aeeba7d71b149a7582a67dd318f6565f08c768ef494d3c: Status 404 returned error can't find the container with id cd698aa8dc7f7badc7aeeba7d71b149a7582a67dd318f6565f08c768ef494d3c Jul 24 00:49:46.034037 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:46.033934 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" event={"ID":"b4b7239e-83fc-49d8-bbf0-8fe569f31bff","Type":"ContainerStarted","Data":"c2fdca71795c7fb0d5b98eaba6a56ae82dbf92bd6a3804614be28f62b49570e2"} Jul 24 00:49:46.034037 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:46.034002 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" event={"ID":"b4b7239e-83fc-49d8-bbf0-8fe569f31bff","Type":"ContainerStarted","Data":"cd698aa8dc7f7badc7aeeba7d71b149a7582a67dd318f6565f08c768ef494d3c"} Jul 24 00:49:46.050049 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:46.050003 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" podStartSLOduration=1.049989478 podStartE2EDuration="1.049989478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:49:46.048732071 +0000 UTC m=+2362.781367096" watchObservedRunningTime="2026-07-24 00:49:46.049989478 +0000 UTC m=+2362.782624501" Jul 24 00:49:47.041716 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:47.041681 2585 generic.go:356] "Generic (PLEG): container finished" podID="b4b7239e-83fc-49d8-bbf0-8fe569f31bff" containerID="c2fdca71795c7fb0d5b98eaba6a56ae82dbf92bd6a3804614be28f62b49570e2" exitCode=0 Jul 24 00:49:47.042099 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:47.041756 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" event={"ID":"b4b7239e-83fc-49d8-bbf0-8fe569f31bff","Type":"ContainerDied","Data":"c2fdca71795c7fb0d5b98eaba6a56ae82dbf92bd6a3804614be28f62b49570e2"} Jul 24 00:49:48.190922 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:48.190900 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:48.257563 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:48.257539 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b7239e-83fc-49d8-bbf0-8fe569f31bff-maas-api-service-ca\") pod \"b4b7239e-83fc-49d8-bbf0-8fe569f31bff\" (UID: \"b4b7239e-83fc-49d8-bbf0-8fe569f31bff\") " Jul 24 00:49:48.257931 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:48.257911 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b7239e-83fc-49d8-bbf0-8fe569f31bff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4b7239e-83fc-49d8-bbf0-8fe569f31bff" (UID: "b4b7239e-83fc-49d8-bbf0-8fe569f31bff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:49:48.363828 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:48.358812 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4b7239e-83fc-49d8-bbf0-8fe569f31bff-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:49:49.054538 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:49.054512 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" Jul 24 00:49:49.054727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:49.054550 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467" event={"ID":"b4b7239e-83fc-49d8-bbf0-8fe569f31bff","Type":"ContainerDied","Data":"cd698aa8dc7f7badc7aeeba7d71b149a7582a67dd318f6565f08c768ef494d3c"} Jul 24 00:49:49.054727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:49:49.054599 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd698aa8dc7f7badc7aeeba7d71b149a7582a67dd318f6565f08c768ef494d3c" Jul 24 00:50:04.934231 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:04.934193 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:50:05.149524 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:05.149495 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467"] Jul 24 00:50:05.153928 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:05.153908 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a4c256-f4467"] Jul 24 00:50:05.800007 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:05.799973 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b7239e-83fc-49d8-bbf0-8fe569f31bff" path="/var/lib/kubelet/pods/b4b7239e-83fc-49d8-bbf0-8fe569f31bff/volumes" Jul 24 00:50:22.333960 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:22.333922 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:50:23.554990 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.554950 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t"] Jul 24 00:50:23.555394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.555377 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b7239e-83fc-49d8-bbf0-8fe569f31bff" containerName="revoke-keys" Jul 24 00:50:23.555394 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.555388 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b7239e-83fc-49d8-bbf0-8fe569f31bff" containerName="revoke-keys" Jul 24 00:50:23.555468 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.555449 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b7239e-83fc-49d8-bbf0-8fe569f31bff" containerName="revoke-keys" Jul 24 00:50:23.559826 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.559808 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.563120 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.563095 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-91a75b-serving-cert\"" Jul 24 00:50:23.595789 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.595761 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t"] Jul 24 00:50:23.674499 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.674465 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68847b5e-e810-4fdd-90c9-726bd6db2b78-openshift-service-ca\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.674717 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.674525 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68847b5e-e810-4fdd-90c9-726bd6db2b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.674717 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.674592 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdhz\" (UniqueName: \"kubernetes.io/projected/68847b5e-e810-4fdd-90c9-726bd6db2b78-kube-api-access-dzdhz\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.775155 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.775126 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68847b5e-e810-4fdd-90c9-726bd6db2b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.775343 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.775194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdhz\" (UniqueName: \"kubernetes.io/projected/68847b5e-e810-4fdd-90c9-726bd6db2b78-kube-api-access-dzdhz\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.775343 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.775258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68847b5e-e810-4fdd-90c9-726bd6db2b78-openshift-service-ca\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.775884 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.775852 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68847b5e-e810-4fdd-90c9-726bd6db2b78-openshift-service-ca\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.777735 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.777716 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-91a75b-serving-cert\"" Jul 24 00:50:23.785194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.785170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdhz\" (UniqueName: \"kubernetes.io/projected/68847b5e-e810-4fdd-90c9-726bd6db2b78-kube-api-access-dzdhz\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.788073 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.788042 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68847b5e-e810-4fdd-90c9-726bd6db2b78-maas-api-tls\") pod \"maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.870181 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.870092 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:23.971017 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.970975 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:50:23.991675 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:23.991652 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:50:24.055678 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:24.055651 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t"] Jul 24 00:50:24.057406 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:50:24.057382 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68847b5e_e810_4fdd_90c9_726bd6db2b78.slice/crio-4fecd95db30fb03fcf6f2d2f1494141715533e12b7c620fc8edfa7c4f21a9f80 WatchSource:0}: Error finding container 4fecd95db30fb03fcf6f2d2f1494141715533e12b7c620fc8edfa7c4f21a9f80: Status 404 returned error can't find the container with id 4fecd95db30fb03fcf6f2d2f1494141715533e12b7c620fc8edfa7c4f21a9f80 Jul 24 00:50:24.058792 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:24.058775 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:50:24.255444 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:24.255363 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" event={"ID":"68847b5e-e810-4fdd-90c9-726bd6db2b78","Type":"ContainerStarted","Data":"4fecd95db30fb03fcf6f2d2f1494141715533e12b7c620fc8edfa7c4f21a9f80"} Jul 24 00:50:25.262672 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:25.262637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" event={"ID":"68847b5e-e810-4fdd-90c9-726bd6db2b78","Type":"ContainerStarted","Data":"0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215"} Jul 24 00:50:25.263080 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:25.262734 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:25.281035 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:25.280986 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" podStartSLOduration=1.8792839620000001 podStartE2EDuration="2.280971112s" totalImagesPullingTime="401.68715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:50:25.279969381 +0000 UTC m=+2402.012604406" watchObservedRunningTime="2026-07-24 00:50:25.280971112 +0000 UTC m=+2402.013606156" Jul 24 00:50:31.274305 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:31.274276 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 00:50:33.534511 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:33.534474 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:50:40.940904 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:50:40.940872 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:51:12.939902 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:51:12.939851 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:51:24.091616 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:51:24.091586 2585 scope.go:117] "RemoveContainer" containerID="78d44d4e661d7cc62c257798d12cf50d6de38ac1643945b6f8d2667357398eab" Jul 24 00:51:29.334640 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:51:29.334605 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:51:47.435299 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:51:47.435268 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:52:05.835725 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:52:05.835690 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:52:20.740442 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:52:20.740365 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:52:36.332454 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:52:36.332418 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:52:52.125255 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:52:52.125225 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:53:06.240659 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:53:06.240625 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:53:23.838124 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:53:23.838089 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:53:36.137777 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:53:36.137737 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:54:08.135916 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:54:08.135879 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:54:23.551632 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:54:23.551594 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:54:24.133173 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:54:24.133142 2585 scope.go:117] "RemoveContainer" containerID="bd24fa695fa00da7488eb865e3615a210e86c1c8b080a41d5d68af010815a8dd" Jul 24 00:54:42.324985 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:54:42.324952 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:55:01.039721 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:01.039682 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:55:15.537122 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:15.537085 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:55:24.016230 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:24.016198 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:55:24.035415 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:24.035391 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 00:55:24.154564 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:24.154534 2585 scope.go:117] "RemoveContainer" containerID="037caf0cc673d1ad7fa9d34b7afb6542f1ca868c829d7fae0fe7c4ea46e995e1" Jul 24 00:55:24.163292 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:24.163274 2585 scope.go:117] "RemoveContainer" containerID="3fa7f281b524c35593bba2960994eb3167b3d5ff5caaccba8b0900c872f4a5c8" Jul 24 00:55:53.533702 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:55:53.533665 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:56:17.731988 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:17.731952 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:56:24.190001 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:24.189971 2585 scope.go:117] "RemoveContainer" containerID="c2fdca71795c7fb0d5b98eaba6a56ae82dbf92bd6a3804614be28f62b49570e2" Jul 24 00:56:32.629169 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:32.629132 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:56:42.386397 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.386362 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-140556977746352"] Jul 24 00:56:42.390077 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.390059 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:42.392721 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.392693 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-g6bng\"" Jul 24 00:56:42.397834 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.397458 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977746352"] Jul 24 00:56:42.505515 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.505483 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghp4s\" (UniqueName: \"kubernetes.io/projected/d868406f-d273-4a9a-b5b0-f6e7ad4ba111-kube-api-access-ghp4s\") pod \"test-curl-7705-140556977746352\" (UID: \"d868406f-d273-4a9a-b5b0-f6e7ad4ba111\") " pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:42.606117 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.606077 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghp4s\" (UniqueName: \"kubernetes.io/projected/d868406f-d273-4a9a-b5b0-f6e7ad4ba111-kube-api-access-ghp4s\") pod \"test-curl-7705-140556977746352\" (UID: \"d868406f-d273-4a9a-b5b0-f6e7ad4ba111\") " pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:42.614471 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.614446 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghp4s\" (UniqueName: \"kubernetes.io/projected/d868406f-d273-4a9a-b5b0-f6e7ad4ba111-kube-api-access-ghp4s\") pod \"test-curl-7705-140556977746352\" (UID: \"d868406f-d273-4a9a-b5b0-f6e7ad4ba111\") " pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:42.700441 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.700362 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:42.836069 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.836043 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977746352"] Jul 24 00:56:42.837927 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:56:42.837900 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd868406f_d273_4a9a_b5b0_f6e7ad4ba111.slice/crio-f26fb5ddcde8df1caa2bf7ad97ea17c5eab1754d171b27fd7a9522f5a27be410 WatchSource:0}: Error finding container f26fb5ddcde8df1caa2bf7ad97ea17c5eab1754d171b27fd7a9522f5a27be410: Status 404 returned error can't find the container with id f26fb5ddcde8df1caa2bf7ad97ea17c5eab1754d171b27fd7a9522f5a27be410 Jul 24 00:56:42.839272 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:42.839254 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 00:56:43.461072 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:43.461034 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977746352" event={"ID":"d868406f-d273-4a9a-b5b0-f6e7ad4ba111","Type":"ContainerStarted","Data":"f26fb5ddcde8df1caa2bf7ad97ea17c5eab1754d171b27fd7a9522f5a27be410"} Jul 24 00:56:44.468181 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:44.468140 2585 generic.go:356] "Generic (PLEG): container finished" podID="d868406f-d273-4a9a-b5b0-f6e7ad4ba111" containerID="568bbb178e4f6584944e0b8b5b67360bbed3ecd762493f314f5967b98d1aa43b" exitCode=0 Jul 24 00:56:44.468559 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:44.468223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977746352" event={"ID":"d868406f-d273-4a9a-b5b0-f6e7ad4ba111","Type":"ContainerDied","Data":"568bbb178e4f6584944e0b8b5b67360bbed3ecd762493f314f5967b98d1aa43b"} Jul 24 00:56:45.612601 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.612577 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:45.654124 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.654054 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7705-140556977746352_d868406f-d273-4a9a-b5b0-f6e7ad4ba111/test-curl-7705-140556977746352/0.log" Jul 24 00:56:45.680408 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.680369 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-140556977746352"] Jul 24 00:56:45.683425 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.683399 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-140556977746352"] Jul 24 00:56:45.737745 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.737662 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghp4s\" (UniqueName: \"kubernetes.io/projected/d868406f-d273-4a9a-b5b0-f6e7ad4ba111-kube-api-access-ghp4s\") pod \"d868406f-d273-4a9a-b5b0-f6e7ad4ba111\" (UID: \"d868406f-d273-4a9a-b5b0-f6e7ad4ba111\") " Jul 24 00:56:45.739771 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.739744 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d868406f-d273-4a9a-b5b0-f6e7ad4ba111-kube-api-access-ghp4s" (OuterVolumeSpecName: "kube-api-access-ghp4s") pod "d868406f-d273-4a9a-b5b0-f6e7ad4ba111" (UID: "d868406f-d273-4a9a-b5b0-f6e7ad4ba111"). InnerVolumeSpecName "kube-api-access-ghp4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:56:45.800589 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.800553 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d868406f-d273-4a9a-b5b0-f6e7ad4ba111" path="/var/lib/kubelet/pods/d868406f-d273-4a9a-b5b0-f6e7ad4ba111/volumes" Jul 24 00:56:45.812362 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.812334 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-140556977664688"] Jul 24 00:56:45.812741 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.812726 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d868406f-d273-4a9a-b5b0-f6e7ad4ba111" containerName="test-curl-7705-140556977746352" Jul 24 00:56:45.812823 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.812743 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d868406f-d273-4a9a-b5b0-f6e7ad4ba111" containerName="test-curl-7705-140556977746352" Jul 24 00:56:45.812911 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.812896 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d868406f-d273-4a9a-b5b0-f6e7ad4ba111" containerName="test-curl-7705-140556977746352" Jul 24 00:56:45.816097 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.816078 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:45.823409 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.823383 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977664688"] Jul 24 00:56:45.839088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.839064 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghp4s\" (UniqueName: \"kubernetes.io/projected/d868406f-d273-4a9a-b5b0-f6e7ad4ba111-kube-api-access-ghp4s\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:56:45.940684 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:45.940637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfv5t\" (UniqueName: \"kubernetes.io/projected/11d167e7-cae2-424e-9732-bba6b3a0e489-kube-api-access-tfv5t\") pod \"test-curl-7705-140556977664688\" (UID: \"11d167e7-cae2-424e-9732-bba6b3a0e489\") " pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:46.041569 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.041479 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfv5t\" (UniqueName: \"kubernetes.io/projected/11d167e7-cae2-424e-9732-bba6b3a0e489-kube-api-access-tfv5t\") pod \"test-curl-7705-140556977664688\" (UID: \"11d167e7-cae2-424e-9732-bba6b3a0e489\") " pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:46.049407 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.049374 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfv5t\" (UniqueName: \"kubernetes.io/projected/11d167e7-cae2-424e-9732-bba6b3a0e489-kube-api-access-tfv5t\") pod \"test-curl-7705-140556977664688\" (UID: \"11d167e7-cae2-424e-9732-bba6b3a0e489\") " pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:46.128198 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.128168 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:46.265618 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.265588 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977664688"] Jul 24 00:56:46.266944 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:56:46.266915 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11d167e7_cae2_424e_9732_bba6b3a0e489.slice/crio-0971f8300194283e81e638f0f744247a2dafc68df2225a10e267420518dd9994 WatchSource:0}: Error finding container 0971f8300194283e81e638f0f744247a2dafc68df2225a10e267420518dd9994: Status 404 returned error can't find the container with id 0971f8300194283e81e638f0f744247a2dafc68df2225a10e267420518dd9994 Jul 24 00:56:46.480874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.480830 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977746352" Jul 24 00:56:46.480874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.480848 2585 scope.go:117] "RemoveContainer" containerID="568bbb178e4f6584944e0b8b5b67360bbed3ecd762493f314f5967b98d1aa43b" Jul 24 00:56:46.482416 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:46.482392 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977664688" event={"ID":"11d167e7-cae2-424e-9732-bba6b3a0e489","Type":"ContainerStarted","Data":"0971f8300194283e81e638f0f744247a2dafc68df2225a10e267420518dd9994"} Jul 24 00:56:47.491057 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:47.490944 2585 generic.go:356] "Generic (PLEG): container finished" podID="11d167e7-cae2-424e-9732-bba6b3a0e489" containerID="f7125f4109c0127434f6c28978ddbdb9be879e86d430b3d8b3914ca5c095fdd0" exitCode=0 Jul 24 00:56:47.491057 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:47.491026 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977664688" event={"ID":"11d167e7-cae2-424e-9732-bba6b3a0e489","Type":"ContainerDied","Data":"f7125f4109c0127434f6c28978ddbdb9be879e86d430b3d8b3914ca5c095fdd0"} Jul 24 00:56:47.742014 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:47.741937 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:56:48.637013 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.636985 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:48.660194 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.660135 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7705-140556977664688_11d167e7-cae2-424e-9732-bba6b3a0e489/test-curl-7705-140556977664688/0.log" Jul 24 00:56:48.686413 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.686382 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-140556977664688"] Jul 24 00:56:48.690163 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.690138 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-140556977664688"] Jul 24 00:56:48.767699 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.767614 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfv5t\" (UniqueName: \"kubernetes.io/projected/11d167e7-cae2-424e-9732-bba6b3a0e489-kube-api-access-tfv5t\") pod \"11d167e7-cae2-424e-9732-bba6b3a0e489\" (UID: \"11d167e7-cae2-424e-9732-bba6b3a0e489\") " Jul 24 00:56:48.769689 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.769663 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d167e7-cae2-424e-9732-bba6b3a0e489-kube-api-access-tfv5t" (OuterVolumeSpecName: "kube-api-access-tfv5t") pod "11d167e7-cae2-424e-9732-bba6b3a0e489" (UID: "11d167e7-cae2-424e-9732-bba6b3a0e489"). InnerVolumeSpecName "kube-api-access-tfv5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:56:48.856754 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.856720 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:48.857166 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.857153 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d167e7-cae2-424e-9732-bba6b3a0e489" containerName="test-curl-7705-140556977664688" Jul 24 00:56:48.857211 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.857167 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d167e7-cae2-424e-9732-bba6b3a0e489" containerName="test-curl-7705-140556977664688" Jul 24 00:56:48.857249 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.857237 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d167e7-cae2-424e-9732-bba6b3a0e489" containerName="test-curl-7705-140556977664688" Jul 24 00:56:48.860403 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.860387 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:48.864390 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.864366 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:48.868709 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.868403 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfv5t\" (UniqueName: \"kubernetes.io/projected/11d167e7-cae2-424e-9732-bba6b3a0e489-kube-api-access-tfv5t\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:56:48.969239 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:48.969200 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmdx\" (UniqueName: \"kubernetes.io/projected/bd0e8303-97b6-44f6-877e-e20a05ab8ad6-kube-api-access-gxmdx\") pod \"test-curl-7705-140556977710256\" (UID: \"bd0e8303-97b6-44f6-877e-e20a05ab8ad6\") " pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:49.070610 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.070573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmdx\" (UniqueName: \"kubernetes.io/projected/bd0e8303-97b6-44f6-877e-e20a05ab8ad6-kube-api-access-gxmdx\") pod \"test-curl-7705-140556977710256\" (UID: \"bd0e8303-97b6-44f6-877e-e20a05ab8ad6\") " pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:49.079123 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.079089 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmdx\" (UniqueName: \"kubernetes.io/projected/bd0e8303-97b6-44f6-877e-e20a05ab8ad6-kube-api-access-gxmdx\") pod \"test-curl-7705-140556977710256\" (UID: \"bd0e8303-97b6-44f6-877e-e20a05ab8ad6\") " pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:49.170924 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.170855 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:49.310958 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.310930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:49.312639 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:56:49.312607 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd0e8303_97b6_44f6_877e_e20a05ab8ad6.slice/crio-35b6f7b5f04e453fa2e58626f9ef42322d16f316941169db17a482d849171f0a WatchSource:0}: Error finding container 35b6f7b5f04e453fa2e58626f9ef42322d16f316941169db17a482d849171f0a: Status 404 returned error can't find the container with id 35b6f7b5f04e453fa2e58626f9ef42322d16f316941169db17a482d849171f0a Jul 24 00:56:49.504040 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.504001 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710256" event={"ID":"bd0e8303-97b6-44f6-877e-e20a05ab8ad6","Type":"ContainerStarted","Data":"35b6f7b5f04e453fa2e58626f9ef42322d16f316941169db17a482d849171f0a"} Jul 24 00:56:49.505316 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.505297 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:49.505316 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.505307 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0971f8300194283e81e638f0f744247a2dafc68df2225a10e267420518dd9994" Jul 24 00:56:49.516033 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:56:49.516004 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-140556977664688\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="11d167e7-cae2-424e-9732-bba6b3a0e489" pod="opendatahub/test-curl-7705-140556977664688" Jul 24 00:56:49.799903 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:49.799853 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d167e7-cae2-424e-9732-bba6b3a0e489" path="/var/lib/kubelet/pods/11d167e7-cae2-424e-9732-bba6b3a0e489/volumes" Jul 24 00:56:50.512732 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:50.512701 2585 generic.go:356] "Generic (PLEG): container finished" podID="bd0e8303-97b6-44f6-877e-e20a05ab8ad6" containerID="bec296ff578d88def2bdd384be49acfd34e4c28bc5a93695b90b318822ff8668" exitCode=0 Jul 24 00:56:50.512935 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:50.512789 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710256" event={"ID":"bd0e8303-97b6-44f6-877e-e20a05ab8ad6","Type":"ContainerDied","Data":"bec296ff578d88def2bdd384be49acfd34e4c28bc5a93695b90b318822ff8668"} Jul 24 00:56:51.654451 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.654423 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:51.674623 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.674592 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7705-140556977710256_bd0e8303-97b6-44f6-877e-e20a05ab8ad6/test-curl-7705-140556977710256/0.log" Jul 24 00:56:51.696336 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.696300 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:51.700402 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.700379 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:51.796542 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.796454 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxmdx\" (UniqueName: \"kubernetes.io/projected/bd0e8303-97b6-44f6-877e-e20a05ab8ad6-kube-api-access-gxmdx\") pod \"bd0e8303-97b6-44f6-877e-e20a05ab8ad6\" (UID: \"bd0e8303-97b6-44f6-877e-e20a05ab8ad6\") " Jul 24 00:56:51.798727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.798695 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0e8303-97b6-44f6-877e-e20a05ab8ad6-kube-api-access-gxmdx" (OuterVolumeSpecName: "kube-api-access-gxmdx") pod "bd0e8303-97b6-44f6-877e-e20a05ab8ad6" (UID: "bd0e8303-97b6-44f6-877e-e20a05ab8ad6"). InnerVolumeSpecName "kube-api-access-gxmdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:56:51.799723 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.799702 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd0e8303-97b6-44f6-877e-e20a05ab8ad6" path="/var/lib/kubelet/pods/bd0e8303-97b6-44f6-877e-e20a05ab8ad6/volumes" Jul 24 00:56:51.824133 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.824104 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:51.824557 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.824539 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd0e8303-97b6-44f6-877e-e20a05ab8ad6" containerName="test-curl-7705-140556977710256" Jul 24 00:56:51.824638 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.824559 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd0e8303-97b6-44f6-877e-e20a05ab8ad6" containerName="test-curl-7705-140556977710256" Jul 24 00:56:51.824698 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.824671 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd0e8303-97b6-44f6-877e-e20a05ab8ad6" containerName="test-curl-7705-140556977710256" Jul 24 00:56:51.827949 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.827932 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:51.831943 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.831924 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:51.897448 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.897411 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmdx\" (UniqueName: \"kubernetes.io/projected/bd0e8303-97b6-44f6-877e-e20a05ab8ad6-kube-api-access-gxmdx\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:56:51.998576 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:51.998537 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkhhv\" (UniqueName: \"kubernetes.io/projected/666914e9-ece8-46ce-8363-24311a7d2562-kube-api-access-bkhhv\") pod \"test-curl-7705-140556977710256\" (UID: \"666914e9-ece8-46ce-8363-24311a7d2562\") " pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:52.099484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.099450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkhhv\" (UniqueName: \"kubernetes.io/projected/666914e9-ece8-46ce-8363-24311a7d2562-kube-api-access-bkhhv\") pod \"test-curl-7705-140556977710256\" (UID: \"666914e9-ece8-46ce-8363-24311a7d2562\") " pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:52.107804 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.107772 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkhhv\" (UniqueName: \"kubernetes.io/projected/666914e9-ece8-46ce-8363-24311a7d2562-kube-api-access-bkhhv\") pod \"test-curl-7705-140556977710256\" (UID: \"666914e9-ece8-46ce-8363-24311a7d2562\") " pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:52.138417 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.138394 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:52.275941 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.275911 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:52.277834 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:56:52.277811 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod666914e9_ece8_46ce_8363_24311a7d2562.slice/crio-21c1a47e22c0d9fe380be6346b9621e62acf69f75598a768bfb656f5a8131cb7 WatchSource:0}: Error finding container 21c1a47e22c0d9fe380be6346b9621e62acf69f75598a768bfb656f5a8131cb7: Status 404 returned error can't find the container with id 21c1a47e22c0d9fe380be6346b9621e62acf69f75598a768bfb656f5a8131cb7 Jul 24 00:56:52.526648 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.526572 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:52.526648 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.526627 2585 scope.go:117] "RemoveContainer" containerID="bec296ff578d88def2bdd384be49acfd34e4c28bc5a93695b90b318822ff8668" Jul 24 00:56:52.527886 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:52.527850 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710256" event={"ID":"666914e9-ece8-46ce-8363-24311a7d2562","Type":"ContainerStarted","Data":"21c1a47e22c0d9fe380be6346b9621e62acf69f75598a768bfb656f5a8131cb7"} Jul 24 00:56:53.535456 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:53.535425 2585 generic.go:356] "Generic (PLEG): container finished" podID="666914e9-ece8-46ce-8363-24311a7d2562" containerID="21a51c149eab6c57c3f1c56131e0a445b42901ee31bce4856d25c2172ba084fb" exitCode=0 Jul 24 00:56:53.535911 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:53.535517 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710256" event={"ID":"666914e9-ece8-46ce-8363-24311a7d2562","Type":"ContainerDied","Data":"21a51c149eab6c57c3f1c56131e0a445b42901ee31bce4856d25c2172ba084fb"} Jul 24 00:56:54.688830 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.688801 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:54.708778 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.708745 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7705-140556977710256_666914e9-ece8-46ce-8363-24311a7d2562/test-curl-7705-140556977710256/0.log" Jul 24 00:56:54.737637 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.737601 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:54.740340 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.740316 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-140556977710256"] Jul 24 00:56:54.823487 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.823462 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkhhv\" (UniqueName: \"kubernetes.io/projected/666914e9-ece8-46ce-8363-24311a7d2562-kube-api-access-bkhhv\") pod \"666914e9-ece8-46ce-8363-24311a7d2562\" (UID: \"666914e9-ece8-46ce-8363-24311a7d2562\") " Jul 24 00:56:54.825489 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.825455 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666914e9-ece8-46ce-8363-24311a7d2562-kube-api-access-bkhhv" (OuterVolumeSpecName: "kube-api-access-bkhhv") pod "666914e9-ece8-46ce-8363-24311a7d2562" (UID: "666914e9-ece8-46ce-8363-24311a7d2562"). InnerVolumeSpecName "kube-api-access-bkhhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:56:54.856654 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.856626 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:54.857056 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.857043 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666914e9-ece8-46ce-8363-24311a7d2562" containerName="test-curl-7705-140556977710256" Jul 24 00:56:54.857105 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.857057 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="666914e9-ece8-46ce-8363-24311a7d2562" containerName="test-curl-7705-140556977710256" Jul 24 00:56:54.857141 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.857133 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="666914e9-ece8-46ce-8363-24311a7d2562" containerName="test-curl-7705-140556977710256" Jul 24 00:56:54.860210 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.860194 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:54.866223 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.866184 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:54.924745 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:54.924716 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkhhv\" (UniqueName: \"kubernetes.io/projected/666914e9-ece8-46ce-8363-24311a7d2562-kube-api-access-bkhhv\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:56:55.026091 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.026057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hvs\" (UniqueName: \"kubernetes.io/projected/9c84a942-ce41-447f-9465-3fcb251b3fda-kube-api-access-c6hvs\") pod \"test-curl-7705-140556977710400\" (UID: \"9c84a942-ce41-447f-9465-3fcb251b3fda\") " pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:55.127684 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.127589 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hvs\" (UniqueName: \"kubernetes.io/projected/9c84a942-ce41-447f-9465-3fcb251b3fda-kube-api-access-c6hvs\") pod \"test-curl-7705-140556977710400\" (UID: \"9c84a942-ce41-447f-9465-3fcb251b3fda\") " pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:55.136684 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.136651 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hvs\" (UniqueName: \"kubernetes.io/projected/9c84a942-ce41-447f-9465-3fcb251b3fda-kube-api-access-c6hvs\") pod \"test-curl-7705-140556977710400\" (UID: \"9c84a942-ce41-447f-9465-3fcb251b3fda\") " pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:55.170727 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.170689 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:55.307286 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.307265 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:55.310189 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:56:55.310161 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c84a942_ce41_447f_9465_3fcb251b3fda.slice/crio-085ca5b4f9859107e5b54c29aa096c898ff04a037e91b2ab527c6f2d88b7c513 WatchSource:0}: Error finding container 085ca5b4f9859107e5b54c29aa096c898ff04a037e91b2ab527c6f2d88b7c513: Status 404 returned error can't find the container with id 085ca5b4f9859107e5b54c29aa096c898ff04a037e91b2ab527c6f2d88b7c513 Jul 24 00:56:55.550003 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.549932 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:55.550003 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.549941 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c1a47e22c0d9fe380be6346b9621e62acf69f75598a768bfb656f5a8131cb7" Jul 24 00:56:55.551277 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.551254 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710400" event={"ID":"9c84a942-ce41-447f-9465-3fcb251b3fda","Type":"ContainerStarted","Data":"085ca5b4f9859107e5b54c29aa096c898ff04a037e91b2ab527c6f2d88b7c513"} Jul 24 00:56:55.560615 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:56:55.560589 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-140556977710256\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="666914e9-ece8-46ce-8363-24311a7d2562" pod="opendatahub/test-curl-7705-140556977710256" Jul 24 00:56:55.800465 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:55.800367 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="666914e9-ece8-46ce-8363-24311a7d2562" path="/var/lib/kubelet/pods/666914e9-ece8-46ce-8363-24311a7d2562/volumes" Jul 24 00:56:56.558732 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:56.558700 2585 generic.go:356] "Generic (PLEG): container finished" podID="9c84a942-ce41-447f-9465-3fcb251b3fda" containerID="485d3cb88463da596183f23018b9eb3addac84beef3b3c4f0d383d553e344fab" exitCode=0 Jul 24 00:56:56.558932 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:56.558749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710400" event={"ID":"9c84a942-ce41-447f-9465-3fcb251b3fda","Type":"ContainerDied","Data":"485d3cb88463da596183f23018b9eb3addac84beef3b3c4f0d383d553e344fab"} Jul 24 00:56:57.715956 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.715934 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:57.735088 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.735064 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7705-140556977710400_9c84a942-ce41-447f-9465-3fcb251b3fda/test-curl-7705-140556977710400/0.log" Jul 24 00:56:57.757381 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.757353 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:57.759680 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.759660 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:57.851587 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.851501 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6hvs\" (UniqueName: \"kubernetes.io/projected/9c84a942-ce41-447f-9465-3fcb251b3fda-kube-api-access-c6hvs\") pod \"9c84a942-ce41-447f-9465-3fcb251b3fda\" (UID: \"9c84a942-ce41-447f-9465-3fcb251b3fda\") " Jul 24 00:56:57.853641 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.853615 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c84a942-ce41-447f-9465-3fcb251b3fda-kube-api-access-c6hvs" (OuterVolumeSpecName: "kube-api-access-c6hvs") pod "9c84a942-ce41-447f-9465-3fcb251b3fda" (UID: "9c84a942-ce41-447f-9465-3fcb251b3fda"). InnerVolumeSpecName "kube-api-access-c6hvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:56:57.893244 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.893206 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:57.893670 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.893652 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c84a942-ce41-447f-9465-3fcb251b3fda" containerName="test-curl-7705-140556977710400" Jul 24 00:56:57.893760 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.893672 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c84a942-ce41-447f-9465-3fcb251b3fda" containerName="test-curl-7705-140556977710400" Jul 24 00:56:57.893818 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.893807 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c84a942-ce41-447f-9465-3fcb251b3fda" containerName="test-curl-7705-140556977710400" Jul 24 00:56:57.896979 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.896959 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:57.903047 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.902917 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:57.952159 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:57.952129 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hvs\" (UniqueName: \"kubernetes.io/projected/9c84a942-ce41-447f-9465-3fcb251b3fda-kube-api-access-c6hvs\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:56:58.053571 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.053539 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8wt\" (UniqueName: \"kubernetes.io/projected/4ecda3cb-deb9-454c-a317-dddce6dca342-kube-api-access-hf8wt\") pod \"test-curl-7705-140556977710400\" (UID: \"4ecda3cb-deb9-454c-a317-dddce6dca342\") " pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:58.154195 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.154111 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8wt\" (UniqueName: \"kubernetes.io/projected/4ecda3cb-deb9-454c-a317-dddce6dca342-kube-api-access-hf8wt\") pod \"test-curl-7705-140556977710400\" (UID: \"4ecda3cb-deb9-454c-a317-dddce6dca342\") " pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:58.162477 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.162453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8wt\" (UniqueName: \"kubernetes.io/projected/4ecda3cb-deb9-454c-a317-dddce6dca342-kube-api-access-hf8wt\") pod \"test-curl-7705-140556977710400\" (UID: \"4ecda3cb-deb9-454c-a317-dddce6dca342\") " pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:58.207400 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.207368 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:58.343083 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.343060 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:56:58.343963 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:56:58.343935 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ecda3cb_deb9_454c_a317_dddce6dca342.slice/crio-8e0791f57b185067afed0d385ebad2c155fc8689dee1e2a906b7191ca2b153d9 WatchSource:0}: Error finding container 8e0791f57b185067afed0d385ebad2c155fc8689dee1e2a906b7191ca2b153d9: Status 404 returned error can't find the container with id 8e0791f57b185067afed0d385ebad2c155fc8689dee1e2a906b7191ca2b153d9 Jul 24 00:56:58.572577 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.572543 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:56:58.572764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.572595 2585 scope.go:117] "RemoveContainer" containerID="485d3cb88463da596183f23018b9eb3addac84beef3b3c4f0d383d553e344fab" Jul 24 00:56:58.573981 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:58.573958 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710400" event={"ID":"4ecda3cb-deb9-454c-a317-dddce6dca342","Type":"ContainerStarted","Data":"8e0791f57b185067afed0d385ebad2c155fc8689dee1e2a906b7191ca2b153d9"} Jul 24 00:56:59.582260 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:59.582222 2585 generic.go:356] "Generic (PLEG): container finished" podID="4ecda3cb-deb9-454c-a317-dddce6dca342" containerID="fb81512feed9a1314c7f5d7a366a4a536a8e26a6b6abfa28ae11458db5228318" exitCode=0 Jul 24 00:56:59.582660 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:59.582310 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-140556977710400" event={"ID":"4ecda3cb-deb9-454c-a317-dddce6dca342","Type":"ContainerDied","Data":"fb81512feed9a1314c7f5d7a366a4a536a8e26a6b6abfa28ae11458db5228318"} Jul 24 00:56:59.800265 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:56:59.800234 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c84a942-ce41-447f-9465-3fcb251b3fda" path="/var/lib/kubelet/pods/9c84a942-ce41-447f-9465-3fcb251b3fda/volumes" Jul 24 00:57:00.735973 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.735949 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:57:00.757096 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.757070 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7705-140556977710400_4ecda3cb-deb9-454c-a317-dddce6dca342/test-curl-7705-140556977710400/0.log" Jul 24 00:57:00.778287 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.778262 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf8wt\" (UniqueName: \"kubernetes.io/projected/4ecda3cb-deb9-454c-a317-dddce6dca342-kube-api-access-hf8wt\") pod \"4ecda3cb-deb9-454c-a317-dddce6dca342\" (UID: \"4ecda3cb-deb9-454c-a317-dddce6dca342\") " Jul 24 00:57:00.780501 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.780468 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecda3cb-deb9-454c-a317-dddce6dca342-kube-api-access-hf8wt" (OuterVolumeSpecName: "kube-api-access-hf8wt") pod "4ecda3cb-deb9-454c-a317-dddce6dca342" (UID: "4ecda3cb-deb9-454c-a317-dddce6dca342"). InnerVolumeSpecName "kube-api-access-hf8wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:57:00.782733 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.782709 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:57:00.785034 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.785014 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-140556977710400"] Jul 24 00:57:00.878986 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:00.878905 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf8wt\" (UniqueName: \"kubernetes.io/projected/4ecda3cb-deb9-454c-a317-dddce6dca342-kube-api-access-hf8wt\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:57:01.594658 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:01.594633 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:57:01.594828 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:01.594671 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0791f57b185067afed0d385ebad2c155fc8689dee1e2a906b7191ca2b153d9" Jul 24 00:57:01.605708 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:57:01.605672 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-140556977710400\" is forbidden: User \"system:node:ip-10-0-136-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-102.ec2.internal' and this object" podUID="4ecda3cb-deb9-454c-a317-dddce6dca342" pod="opendatahub/test-curl-7705-140556977710400" Jul 24 00:57:01.801425 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:01.801392 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ecda3cb-deb9-454c-a317-dddce6dca342" path="/var/lib/kubelet/pods/4ecda3cb-deb9-454c-a317-dddce6dca342/volumes" Jul 24 00:57:12.941313 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:12.941274 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:57:23.343917 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.343852 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq"] Jul 24 00:57:23.344421 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.344388 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecda3cb-deb9-454c-a317-dddce6dca342" containerName="test-curl-7705-140556977710400" Jul 24 00:57:23.344421 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.344409 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecda3cb-deb9-454c-a317-dddce6dca342" containerName="test-curl-7705-140556977710400" Jul 24 00:57:23.344572 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.344533 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ecda3cb-deb9-454c-a317-dddce6dca342" containerName="test-curl-7705-140556977710400" Jul 24 00:57:23.349344 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.349316 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.352221 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.352195 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4fa1d5-serving-cert\"" Jul 24 00:57:23.360055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.360033 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq"] Jul 24 00:57:23.478947 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.478909 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78da9f13-4e75-432e-ae55-85f20769d8d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.478947 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.478949 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.479168 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.479075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9sjf\" (UniqueName: \"kubernetes.io/projected/78da9f13-4e75-432e-ae55-85f20769d8d2-kube-api-access-h9sjf\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.580597 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.580550 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9sjf\" (UniqueName: \"kubernetes.io/projected/78da9f13-4e75-432e-ae55-85f20769d8d2-kube-api-access-h9sjf\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.580790 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.580664 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78da9f13-4e75-432e-ae55-85f20769d8d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.580790 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.580691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.580944 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:57:23.580815 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-serving-cert: secret "maas-api-e2e-ipp-b-4fa1d5-serving-cert" not found Jul 24 00:57:23.580944 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:57:23.580899 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls podName:78da9f13-4e75-432e-ae55-85f20769d8d2 nodeName:}" failed. No retries permitted until 2026-07-24 00:57:24.0808791 +0000 UTC m=+2820.813514119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls") pod "maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" (UID: "78da9f13-4e75-432e-ae55-85f20769d8d2") : secret "maas-api-e2e-ipp-b-4fa1d5-serving-cert" not found Jul 24 00:57:23.581395 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.581368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78da9f13-4e75-432e-ae55-85f20769d8d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:23.593562 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:23.593536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9sjf\" (UniqueName: \"kubernetes.io/projected/78da9f13-4e75-432e-ae55-85f20769d8d2-kube-api-access-h9sjf\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:24.085481 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:24.085446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:24.087942 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:24.087917 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:24.260911 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:24.260885 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:24.418280 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:24.418248 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq"] Jul 24 00:57:24.420035 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:57:24.420000 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78da9f13_4e75_432e_ae55_85f20769d8d2.slice/crio-5b4014abb7e8780d9640032aa758d4cafafc3afee5117e77d7dd7091fff03b9a WatchSource:0}: Error finding container 5b4014abb7e8780d9640032aa758d4cafafc3afee5117e77d7dd7091fff03b9a: Status 404 returned error can't find the container with id 5b4014abb7e8780d9640032aa758d4cafafc3afee5117e77d7dd7091fff03b9a Jul 24 00:57:24.737041 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:24.736956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" event={"ID":"78da9f13-4e75-432e-ae55-85f20769d8d2","Type":"ContainerStarted","Data":"5b4014abb7e8780d9640032aa758d4cafafc3afee5117e77d7dd7091fff03b9a"} Jul 24 00:57:25.745687 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:25.745643 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" event={"ID":"78da9f13-4e75-432e-ae55-85f20769d8d2","Type":"ContainerStarted","Data":"2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113"} Jul 24 00:57:25.746182 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:25.745738 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:25.772661 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:25.772600 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" podStartSLOduration=2.435991059 podStartE2EDuration="2.772580451s" totalImagesPullingTime="336.589392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:57:25.769303916 +0000 UTC m=+2822.501938952" watchObservedRunningTime="2026-07-24 00:57:25.772580451 +0000 UTC m=+2822.505215477" Jul 24 00:57:31.759223 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:31.759194 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 00:57:32.229919 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:32.229888 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:57:44.882756 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:44.882712 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x24f_262c57a9-2608-4235-bb94-b67ac50ac889/payload-processing/0.log" Jul 24 00:57:57.536808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:57:57.536774 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:58:20.727368 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:20.727340 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:58:32.015339 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:32.015304 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x24f_262c57a9-2608-4235-bb94-b67ac50ac889/payload-processing/0.log" Jul 24 00:58:36.036165 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.036129 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h"] Jul 24 00:58:36.039925 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.039905 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.042925 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.042903 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-78d793-serving-cert\"" Jul 24 00:58:36.047278 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.047254 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.047426 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.047407 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c600386a-c9b8-4460-87ec-20e10d25a632-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.047504 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.047457 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9m5h\" (UniqueName: \"kubernetes.io/projected/c600386a-c9b8-4460-87ec-20e10d25a632-kube-api-access-n9m5h\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.055221 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.055196 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h"] Jul 24 00:58:36.148175 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.148139 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c600386a-c9b8-4460-87ec-20e10d25a632-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.148381 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.148187 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9m5h\" (UniqueName: \"kubernetes.io/projected/c600386a-c9b8-4460-87ec-20e10d25a632-kube-api-access-n9m5h\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.148381 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.148226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.148381 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:58:36.148334 2585 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-serving-cert: secret "maas-api-e2e-ipp-cleanup-78d793-serving-cert" not found Jul 24 00:58:36.148567 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:58:36.148425 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls podName:c600386a-c9b8-4460-87ec-20e10d25a632 nodeName:}" failed. No retries permitted until 2026-07-24 00:58:36.648404615 +0000 UTC m=+2893.381039635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" (UID: "c600386a-c9b8-4460-87ec-20e10d25a632") : secret "maas-api-e2e-ipp-cleanup-78d793-serving-cert" not found Jul 24 00:58:36.148815 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.148794 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c600386a-c9b8-4460-87ec-20e10d25a632-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.158764 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.158730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9m5h\" (UniqueName: \"kubernetes.io/projected/c600386a-c9b8-4460-87ec-20e10d25a632-kube-api-access-n9m5h\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.651307 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.651277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.653792 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.653764 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:36.950484 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:36.950390 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:37.098937 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:37.098905 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h"] Jul 24 00:58:37.100134 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:58:37.100106 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc600386a_c9b8_4460_87ec_20e10d25a632.slice/crio-4ffd56d1e90b26d94c6994be492e6ef72c7e7553a037a695b78daadfc2d4cc83 WatchSource:0}: Error finding container 4ffd56d1e90b26d94c6994be492e6ef72c7e7553a037a695b78daadfc2d4cc83: Status 404 returned error can't find the container with id 4ffd56d1e90b26d94c6994be492e6ef72c7e7553a037a695b78daadfc2d4cc83 Jul 24 00:58:37.163676 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:37.163633 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" event={"ID":"c600386a-c9b8-4460-87ec-20e10d25a632","Type":"ContainerStarted","Data":"4ffd56d1e90b26d94c6994be492e6ef72c7e7553a037a695b78daadfc2d4cc83"} Jul 24 00:58:38.171674 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:38.171632 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" event={"ID":"c600386a-c9b8-4460-87ec-20e10d25a632","Type":"ContainerStarted","Data":"71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93"} Jul 24 00:58:38.172149 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:38.171731 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:38.195424 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:38.195364 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" podStartSLOduration=1.854289175 podStartE2EDuration="2.195345759s" totalImagesPullingTime="341.056584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:38.193420077 +0000 UTC m=+2894.926055105" watchObservedRunningTime="2026-07-24 00:58:38.195345759 +0000 UTC m=+2894.927980785" Jul 24 00:58:41.532801 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:41.532768 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:58:44.183359 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:44.183333 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:58:51.417808 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.417771 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65"] Jul 24 00:58:51.423010 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.422988 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:51.426427 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.426404 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:58:51.427898 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.427853 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65"] Jul 24 00:58:51.490282 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.490247 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecf9dca-4b11-475c-9299-258554b35431-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65\" (UID: \"1ecf9dca-4b11-475c-9299-258554b35431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:51.590941 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.590904 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecf9dca-4b11-475c-9299-258554b35431-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65\" (UID: \"1ecf9dca-4b11-475c-9299-258554b35431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:51.591545 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.591511 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecf9dca-4b11-475c-9299-258554b35431-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65\" (UID: \"1ecf9dca-4b11-475c-9299-258554b35431\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:51.737944 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.737827 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:51.878376 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:51.878347 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65"] Jul 24 00:58:51.884528 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:58:51.883761 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecf9dca_4b11_475c_9299_258554b35431.slice/crio-00eb9748cd0ddc5681f81b2dbdae75762f2a3008de07dd0f1fbb57c75dfc3211 WatchSource:0}: Error finding container 00eb9748cd0ddc5681f81b2dbdae75762f2a3008de07dd0f1fbb57c75dfc3211: Status 404 returned error can't find the container with id 00eb9748cd0ddc5681f81b2dbdae75762f2a3008de07dd0f1fbb57c75dfc3211 Jul 24 00:58:52.261874 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:52.261818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" event={"ID":"1ecf9dca-4b11-475c-9299-258554b35431","Type":"ContainerStarted","Data":"5f2e9106b10080039fc621397a4b8f99e0271daab8db734930df73ee7ccf5988"} Jul 24 00:58:52.262057 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:52.261899 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" event={"ID":"1ecf9dca-4b11-475c-9299-258554b35431","Type":"ContainerStarted","Data":"00eb9748cd0ddc5681f81b2dbdae75762f2a3008de07dd0f1fbb57c75dfc3211"} Jul 24 00:58:52.278104 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:52.278058 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" podStartSLOduration=1.27804352 podStartE2EDuration="1.27804352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:58:52.276692262 +0000 UTC m=+2909.009327292" watchObservedRunningTime="2026-07-24 00:58:52.27804352 +0000 UTC m=+2909.010678581" Jul 24 00:58:53.269729 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:53.269694 2585 generic.go:356] "Generic (PLEG): container finished" podID="1ecf9dca-4b11-475c-9299-258554b35431" containerID="5f2e9106b10080039fc621397a4b8f99e0271daab8db734930df73ee7ccf5988" exitCode=0 Jul 24 00:58:53.270213 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:53.269782 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" event={"ID":"1ecf9dca-4b11-475c-9299-258554b35431","Type":"ContainerDied","Data":"5f2e9106b10080039fc621397a4b8f99e0271daab8db734930df73ee7ccf5988"} Jul 24 00:58:54.420131 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:54.420106 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:54.516456 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:54.516408 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecf9dca-4b11-475c-9299-258554b35431-maas-api-service-ca\") pod \"1ecf9dca-4b11-475c-9299-258554b35431\" (UID: \"1ecf9dca-4b11-475c-9299-258554b35431\") " Jul 24 00:58:54.516766 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:54.516743 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecf9dca-4b11-475c-9299-258554b35431-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ecf9dca-4b11-475c-9299-258554b35431" (UID: "1ecf9dca-4b11-475c-9299-258554b35431"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:58:54.617710 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:54.617674 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ecf9dca-4b11-475c-9299-258554b35431-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:58:55.287082 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:55.287046 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" event={"ID":"1ecf9dca-4b11-475c-9299-258554b35431","Type":"ContainerDied","Data":"00eb9748cd0ddc5681f81b2dbdae75762f2a3008de07dd0f1fbb57c75dfc3211"} Jul 24 00:58:55.287082 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:55.287077 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65" Jul 24 00:58:55.287349 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:58:55.287097 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00eb9748cd0ddc5681f81b2dbdae75762f2a3008de07dd0f1fbb57c75dfc3211" Jul 24 00:59:01.966848 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:01.966806 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h"] Jul 24 00:59:01.967335 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:01.967077 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" podUID="c600386a-c9b8-4460-87ec-20e10d25a632" containerName="maas-api" containerID="cri-o://71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93" gracePeriod=30 Jul 24 00:59:02.232502 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.232449 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:59:02.294082 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.294051 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c600386a-c9b8-4460-87ec-20e10d25a632-openshift-service-ca\") pod \"c600386a-c9b8-4460-87ec-20e10d25a632\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " Jul 24 00:59:02.294236 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.294114 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9m5h\" (UniqueName: \"kubernetes.io/projected/c600386a-c9b8-4460-87ec-20e10d25a632-kube-api-access-n9m5h\") pod \"c600386a-c9b8-4460-87ec-20e10d25a632\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " Jul 24 00:59:02.294236 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.294198 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls\") pod \"c600386a-c9b8-4460-87ec-20e10d25a632\" (UID: \"c600386a-c9b8-4460-87ec-20e10d25a632\") " Jul 24 00:59:02.294465 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.294442 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c600386a-c9b8-4460-87ec-20e10d25a632-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c600386a-c9b8-4460-87ec-20e10d25a632" (UID: "c600386a-c9b8-4460-87ec-20e10d25a632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:59:02.296333 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.296300 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c600386a-c9b8-4460-87ec-20e10d25a632" (UID: "c600386a-c9b8-4460-87ec-20e10d25a632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 00:59:02.296446 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.296366 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c600386a-c9b8-4460-87ec-20e10d25a632-kube-api-access-n9m5h" (OuterVolumeSpecName: "kube-api-access-n9m5h") pod "c600386a-c9b8-4460-87ec-20e10d25a632" (UID: "c600386a-c9b8-4460-87ec-20e10d25a632"). InnerVolumeSpecName "kube-api-access-n9m5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 00:59:02.329823 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.329795 2585 generic.go:356] "Generic (PLEG): container finished" podID="c600386a-c9b8-4460-87ec-20e10d25a632" containerID="71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93" exitCode=0 Jul 24 00:59:02.329976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.329853 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" Jul 24 00:59:02.329976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.329905 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" event={"ID":"c600386a-c9b8-4460-87ec-20e10d25a632","Type":"ContainerDied","Data":"71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93"} Jul 24 00:59:02.329976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.329956 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h" event={"ID":"c600386a-c9b8-4460-87ec-20e10d25a632","Type":"ContainerDied","Data":"4ffd56d1e90b26d94c6994be492e6ef72c7e7553a037a695b78daadfc2d4cc83"} Jul 24 00:59:02.330129 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.329979 2585 scope.go:117] "RemoveContainer" containerID="71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93" Jul 24 00:59:02.340245 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.340230 2585 scope.go:117] "RemoveContainer" containerID="71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93" Jul 24 00:59:02.340520 ip-10-0-136-102 kubenswrapper[2585]: E0724 00:59:02.340504 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93\": container with ID starting with 71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93 not found: ID does not exist" containerID="71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93" Jul 24 00:59:02.340564 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.340527 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93"} err="failed to get container status \"71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93\": rpc error: code = NotFound desc = could not find container \"71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93\": container with ID starting with 71c3de5a9f28188cc8781cbab8dda21b712f863102e294bbacf8b57a51f92d93 not found: ID does not exist" Jul 24 00:59:02.353626 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.353603 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h"] Jul 24 00:59:02.358480 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.358461 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-78d793-54f99dc4cc-jqd4h"] Jul 24 00:59:02.395291 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.395262 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c600386a-c9b8-4460-87ec-20e10d25a632-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:59:02.395291 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.395288 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c600386a-c9b8-4460-87ec-20e10d25a632-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:59:02.395431 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:02.395302 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9m5h\" (UniqueName: \"kubernetes.io/projected/c600386a-c9b8-4460-87ec-20e10d25a632-kube-api-access-n9m5h\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:59:03.800794 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:03.800758 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c600386a-c9b8-4460-87ec-20e10d25a632" path="/var/lib/kubelet/pods/c600386a-c9b8-4460-87ec-20e10d25a632/volumes" Jul 24 00:59:06.498216 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:06.498165 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65"] Jul 24 00:59:06.500905 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:06.500881 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-78d793-jpr65"] Jul 24 00:59:06.735577 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:06.735544 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:59:07.799666 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:07.799631 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ecf9dca-4b11-475c-9299-258554b35431" path="/var/lib/kubelet/pods/1ecf9dca-4b11-475c-9299-258554b35431/volumes" Jul 24 00:59:16.600080 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600035 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj"] Jul 24 00:59:16.600705 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600686 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c600386a-c9b8-4460-87ec-20e10d25a632" containerName="maas-api" Jul 24 00:59:16.600749 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600708 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c600386a-c9b8-4460-87ec-20e10d25a632" containerName="maas-api" Jul 24 00:59:16.600749 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600722 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ecf9dca-4b11-475c-9299-258554b35431" containerName="revoke-keys" Jul 24 00:59:16.600749 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600730 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ecf9dca-4b11-475c-9299-258554b35431" containerName="revoke-keys" Jul 24 00:59:16.600875 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600848 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c600386a-c9b8-4460-87ec-20e10d25a632" containerName="maas-api" Jul 24 00:59:16.600933 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.600880 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ecf9dca-4b11-475c-9299-258554b35431" containerName="revoke-keys" Jul 24 00:59:16.605657 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.605633 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:16.608474 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.608438 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 00:59:16.609467 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.609440 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj"] Jul 24 00:59:16.722268 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.722232 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c064ab23-fd69-4b30-a361-c85c747f2fdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj\" (UID: \"c064ab23-fd69-4b30-a361-c85c747f2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:16.823152 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.823118 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c064ab23-fd69-4b30-a361-c85c747f2fdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj\" (UID: \"c064ab23-fd69-4b30-a361-c85c747f2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:16.823738 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.823719 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c064ab23-fd69-4b30-a361-c85c747f2fdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj\" (UID: \"c064ab23-fd69-4b30-a361-c85c747f2fdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:16.918922 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:16.918839 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:17.067370 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:17.067339 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj"] Jul 24 00:59:17.068395 ip-10-0-136-102 kubenswrapper[2585]: W0724 00:59:17.068370 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc064ab23_fd69_4b30_a361_c85c747f2fdf.slice/crio-29ba618e3fd5ffe422f6d5a3b174e02e5b7fb4c01c90d81b597f16e79b998831 WatchSource:0}: Error finding container 29ba618e3fd5ffe422f6d5a3b174e02e5b7fb4c01c90d81b597f16e79b998831: Status 404 returned error can't find the container with id 29ba618e3fd5ffe422f6d5a3b174e02e5b7fb4c01c90d81b597f16e79b998831 Jul 24 00:59:17.426055 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:17.426019 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" event={"ID":"c064ab23-fd69-4b30-a361-c85c747f2fdf","Type":"ContainerStarted","Data":"97f0391559151625baea47ace7e6ff0adfd3fe6a55e5ebfd5d65d0fdfc08c295"} Jul 24 00:59:17.426216 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:17.426082 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" event={"ID":"c064ab23-fd69-4b30-a361-c85c747f2fdf","Type":"ContainerStarted","Data":"29ba618e3fd5ffe422f6d5a3b174e02e5b7fb4c01c90d81b597f16e79b998831"} Jul 24 00:59:17.442229 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:17.442185 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" podStartSLOduration=1.442170824 podStartE2EDuration="1.442170824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 00:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 00:59:17.440065385 +0000 UTC m=+2934.172700410" watchObservedRunningTime="2026-07-24 00:59:17.442170824 +0000 UTC m=+2934.174805848" Jul 24 00:59:18.434026 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:18.433987 2585 generic.go:356] "Generic (PLEG): container finished" podID="c064ab23-fd69-4b30-a361-c85c747f2fdf" containerID="97f0391559151625baea47ace7e6ff0adfd3fe6a55e5ebfd5d65d0fdfc08c295" exitCode=0 Jul 24 00:59:18.434397 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:18.434068 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" event={"ID":"c064ab23-fd69-4b30-a361-c85c747f2fdf","Type":"ContainerDied","Data":"97f0391559151625baea47ace7e6ff0adfd3fe6a55e5ebfd5d65d0fdfc08c295"} Jul 24 00:59:19.580997 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:19.580972 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:19.752841 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:19.752761 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c064ab23-fd69-4b30-a361-c85c747f2fdf-maas-api-service-ca\") pod \"c064ab23-fd69-4b30-a361-c85c747f2fdf\" (UID: \"c064ab23-fd69-4b30-a361-c85c747f2fdf\") " Jul 24 00:59:19.753150 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:19.753127 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c064ab23-fd69-4b30-a361-c85c747f2fdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c064ab23-fd69-4b30-a361-c85c747f2fdf" (UID: "c064ab23-fd69-4b30-a361-c85c747f2fdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 00:59:19.853505 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:19.853480 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c064ab23-fd69-4b30-a361-c85c747f2fdf-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 00:59:20.446279 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:20.446246 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" event={"ID":"c064ab23-fd69-4b30-a361-c85c747f2fdf","Type":"ContainerDied","Data":"29ba618e3fd5ffe422f6d5a3b174e02e5b7fb4c01c90d81b597f16e79b998831"} Jul 24 00:59:20.446453 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:20.446296 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29ba618e3fd5ffe422f6d5a3b174e02e5b7fb4c01c90d81b597f16e79b998831" Jul 24 00:59:20.446453 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:20.446255 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj" Jul 24 00:59:31.434129 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:31.434090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 00:59:36.680284 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:36.680246 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj"] Jul 24 00:59:36.684773 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:36.684750 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-193ff5-xscjj"] Jul 24 00:59:37.801310 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:37.801274 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c064ab23-fd69-4b30-a361-c85c747f2fdf" path="/var/lib/kubelet/pods/c064ab23-fd69-4b30-a361-c85c747f2fdf/volumes" Jul 24 00:59:42.631976 ip-10-0-136-102 kubenswrapper[2585]: I0724 00:59:42.631943 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 01:00:00.139598 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.139548 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb"] Jul 24 01:00:00.140267 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.140206 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c064ab23-fd69-4b30-a361-c85c747f2fdf" containerName="revoke-keys" Jul 24 01:00:00.140267 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.140228 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c064ab23-fd69-4b30-a361-c85c747f2fdf" containerName="revoke-keys" Jul 24 01:00:00.140487 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.140347 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c064ab23-fd69-4b30-a361-c85c747f2fdf" containerName="revoke-keys" Jul 24 01:00:00.143896 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.143875 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss"] Jul 24 01:00:00.144011 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.143975 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" Jul 24 01:00:00.146464 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.146444 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 01:00:00.147519 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.147500 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp"] Jul 24 01:00:00.147631 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.147616 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" Jul 24 01:00:00.152396 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.151655 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s"] Jul 24 01:00:00.154768 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.153005 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" Jul 24 01:00:00.157736 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.157716 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb"] Jul 24 01:00:00.157892 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.157820 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" Jul 24 01:00:00.162180 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.162163 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" Jul 24 01:00:00.162539 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.162515 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp"] Jul 24 01:00:00.166158 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.166142 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s"] Jul 24 01:00:00.169852 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.169827 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss"] Jul 24 01:00:00.177401 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.177384 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" Jul 24 01:00:00.226790 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.226749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" Jul 24 01:00:00.233564 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.233544 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" Jul 24 01:00:00.338607 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.338566 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb"] Jul 24 01:00:00.340234 ip-10-0-136-102 kubenswrapper[2585]: W0724 01:00:00.340186 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c672b2_70ad_44ae_99b5_0e273c42908a.slice/crio-fafe818deac1bcda290f61f64a1577b3034a8b9791ae809de851ca5711b74d2a WatchSource:0}: Error finding container fafe818deac1bcda290f61f64a1577b3034a8b9791ae809de851ca5711b74d2a: Status 404 returned error can't find the container with id fafe818deac1bcda290f61f64a1577b3034a8b9791ae809de851ca5711b74d2a Jul 24 01:00:00.379978 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.379951 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss"] Jul 24 01:00:00.381875 ip-10-0-136-102 kubenswrapper[2585]: W0724 01:00:00.381843 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0643c6f3_b637_4ea4_982d_08af86ce1ff7.slice/crio-47a841f51e08451ac58f7d3a274431bc91df1a6a009d37bcae9c843e9e51bbc6 WatchSource:0}: Error finding container 47a841f51e08451ac58f7d3a274431bc91df1a6a009d37bcae9c843e9e51bbc6: Status 404 returned error can't find the container with id 47a841f51e08451ac58f7d3a274431bc91df1a6a009d37bcae9c843e9e51bbc6 Jul 24 01:00:00.417335 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.417313 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp"] Jul 24 01:00:00.420120 ip-10-0-136-102 kubenswrapper[2585]: W0724 01:00:00.420096 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09982e89_a288_45b7_8e99_f6fb8c0a79b2.slice/crio-2a8fbb5f93d34591bc047cda994ef6ae4971cd57a5b4b1e66bfb43240b7e71a0 WatchSource:0}: Error finding container 2a8fbb5f93d34591bc047cda994ef6ae4971cd57a5b4b1e66bfb43240b7e71a0: Status 404 returned error can't find the container with id 2a8fbb5f93d34591bc047cda994ef6ae4971cd57a5b4b1e66bfb43240b7e71a0 Jul 24 01:00:00.437710 ip-10-0-136-102 kubenswrapper[2585]: W0724 01:00:00.437686 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f65f92_4178_44ab_8aea_efa9de9e4a46.slice/crio-bdc7bcb359134b6ef13c07889d0f5d46b7be1233ba3672059cae8f24bcb9e765 WatchSource:0}: Error finding container bdc7bcb359134b6ef13c07889d0f5d46b7be1233ba3672059cae8f24bcb9e765: Status 404 returned error can't find the container with id bdc7bcb359134b6ef13c07889d0f5d46b7be1233ba3672059cae8f24bcb9e765 Jul 24 01:00:00.438936 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.438912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s"] Jul 24 01:00:00.692924 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.692893 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" event={"ID":"b4c672b2-70ad-44ae-99b5-0e273c42908a","Type":"ContainerStarted","Data":"fafe818deac1bcda290f61f64a1577b3034a8b9791ae809de851ca5711b74d2a"} Jul 24 01:00:00.694028 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.694006 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" event={"ID":"09982e89-a288-45b7-8e99-f6fb8c0a79b2","Type":"ContainerStarted","Data":"2a8fbb5f93d34591bc047cda994ef6ae4971cd57a5b4b1e66bfb43240b7e71a0"} Jul 24 01:00:00.695481 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.695458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" event={"ID":"a4f65f92-4178-44ab-8aea-efa9de9e4a46","Type":"ContainerStarted","Data":"bdc7bcb359134b6ef13c07889d0f5d46b7be1233ba3672059cae8f24bcb9e765"} Jul 24 01:00:00.696839 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.696817 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" event={"ID":"0643c6f3-b637-4ea4-982d-08af86ce1ff7","Type":"ContainerStarted","Data":"2e3f78fce08cb42365e869c07fa94fc8611a08ae380ac6c397437e2d5dd23b8c"} Jul 24 01:00:00.696972 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.696854 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" event={"ID":"0643c6f3-b637-4ea4-982d-08af86ce1ff7","Type":"ContainerStarted","Data":"47a841f51e08451ac58f7d3a274431bc91df1a6a009d37bcae9c843e9e51bbc6"} Jul 24 01:00:00.710386 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.710346 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" podStartSLOduration=0.71033306 podStartE2EDuration="710.33306ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:00.708561903 +0000 UTC m=+2977.441196928" watchObservedRunningTime="2026-07-24 01:00:00.71033306 +0000 UTC m=+2977.442968076" Jul 24 01:00:00.725171 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:00.725126 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" podStartSLOduration=0.725110458 podStartE2EDuration="725.110458ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:00:00.721972149 +0000 UTC m=+2977.454607177" watchObservedRunningTime="2026-07-24 01:00:00.725110458 +0000 UTC m=+2977.457745525" Jul 24 01:00:01.703802 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.703766 2585 generic.go:356] "Generic (PLEG): container finished" podID="a4f65f92-4178-44ab-8aea-efa9de9e4a46" containerID="905a72921277d2ac06f882bf48a9c2edf350ed9c417bfbdf5e673e9980b4055b" exitCode=0 Jul 24 01:00:01.704233 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.703845 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" event={"ID":"a4f65f92-4178-44ab-8aea-efa9de9e4a46","Type":"ContainerDied","Data":"905a72921277d2ac06f882bf48a9c2edf350ed9c417bfbdf5e673e9980b4055b"} Jul 24 01:00:01.705612 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.705590 2585 generic.go:356] "Generic (PLEG): container finished" podID="0643c6f3-b637-4ea4-982d-08af86ce1ff7" containerID="2e3f78fce08cb42365e869c07fa94fc8611a08ae380ac6c397437e2d5dd23b8c" exitCode=0 Jul 24 01:00:01.705710 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.705660 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" event={"ID":"0643c6f3-b637-4ea4-982d-08af86ce1ff7","Type":"ContainerDied","Data":"2e3f78fce08cb42365e869c07fa94fc8611a08ae380ac6c397437e2d5dd23b8c"} Jul 24 01:00:01.707230 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.707210 2585 generic.go:356] "Generic (PLEG): container finished" podID="b4c672b2-70ad-44ae-99b5-0e273c42908a" containerID="0f816a207ca1348361bef39d6416eed9dfdb2b10114c851c7b987deafd70f60b" exitCode=0 Jul 24 01:00:01.707344 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.707277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" event={"ID":"b4c672b2-70ad-44ae-99b5-0e273c42908a","Type":"ContainerDied","Data":"0f816a207ca1348361bef39d6416eed9dfdb2b10114c851c7b987deafd70f60b"} Jul 24 01:00:01.708635 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.708611 2585 generic.go:356] "Generic (PLEG): container finished" podID="09982e89-a288-45b7-8e99-f6fb8c0a79b2" containerID="5fe59436eaa118910f24798d495e29b0fb4fccaa414e45d8464bc44a95e2d0e8" exitCode=0 Jul 24 01:00:01.708732 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:01.708641 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" event={"ID":"09982e89-a288-45b7-8e99-f6fb8c0a79b2","Type":"ContainerDied","Data":"5fe59436eaa118910f24798d495e29b0fb4fccaa414e45d8464bc44a95e2d0e8"} Jul 24 01:00:02.857049 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:02.857027 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" Jul 24 01:00:02.904106 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:02.904081 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" Jul 24 01:00:02.943260 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:02.943238 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" Jul 24 01:00:02.947113 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:02.947088 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" Jul 24 01:00:03.722616 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.722590 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" Jul 24 01:00:03.722616 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.722601 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp" event={"ID":"09982e89-a288-45b7-8e99-f6fb8c0a79b2","Type":"ContainerDied","Data":"2a8fbb5f93d34591bc047cda994ef6ae4971cd57a5b4b1e66bfb43240b7e71a0"} Jul 24 01:00:03.722888 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.722652 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8fbb5f93d34591bc047cda994ef6ae4971cd57a5b4b1e66bfb43240b7e71a0" Jul 24 01:00:03.723940 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.723924 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" Jul 24 01:00:03.724081 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.723979 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747580-dw95s" event={"ID":"a4f65f92-4178-44ab-8aea-efa9de9e4a46","Type":"ContainerDied","Data":"bdc7bcb359134b6ef13c07889d0f5d46b7be1233ba3672059cae8f24bcb9e765"} Jul 24 01:00:03.724081 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.724022 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc7bcb359134b6ef13c07889d0f5d46b7be1233ba3672059cae8f24bcb9e765" Jul 24 01:00:03.725339 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.725317 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" Jul 24 01:00:03.725449 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.725358 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss" event={"ID":"0643c6f3-b637-4ea4-982d-08af86ce1ff7","Type":"ContainerDied","Data":"47a841f51e08451ac58f7d3a274431bc91df1a6a009d37bcae9c843e9e51bbc6"} Jul 24 01:00:03.725449 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.725396 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a841f51e08451ac58f7d3a274431bc91df1a6a009d37bcae9c843e9e51bbc6" Jul 24 01:00:03.726791 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.726770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" event={"ID":"b4c672b2-70ad-44ae-99b5-0e273c42908a","Type":"ContainerDied","Data":"fafe818deac1bcda290f61f64a1577b3034a8b9791ae809de851ca5711b74d2a"} Jul 24 01:00:03.726893 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.726800 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb" Jul 24 01:00:03.726893 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:03.726809 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fafe818deac1bcda290f61f64a1577b3034a8b9791ae809de851ca5711b74d2a" Jul 24 01:00:24.061206 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:24.061173 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 01:00:24.079038 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:24.079013 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 01:00:25.342282 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342255 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf"] Jul 24 01:00:25.342683 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342670 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f65f92-4178-44ab-8aea-efa9de9e4a46" containerName="cleanup" Jul 24 01:00:25.342727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342684 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f65f92-4178-44ab-8aea-efa9de9e4a46" containerName="cleanup" Jul 24 01:00:25.342727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342696 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c672b2-70ad-44ae-99b5-0e273c42908a" containerName="cleanup" Jul 24 01:00:25.342727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342703 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c672b2-70ad-44ae-99b5-0e273c42908a" containerName="cleanup" Jul 24 01:00:25.342727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342711 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0643c6f3-b637-4ea4-982d-08af86ce1ff7" containerName="cleanup" Jul 24 01:00:25.342727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342717 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0643c6f3-b637-4ea4-982d-08af86ce1ff7" containerName="cleanup" Jul 24 01:00:25.342900 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342733 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09982e89-a288-45b7-8e99-f6fb8c0a79b2" containerName="cleanup" Jul 24 01:00:25.342900 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342738 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="09982e89-a288-45b7-8e99-f6fb8c0a79b2" containerName="cleanup" Jul 24 01:00:25.342900 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342803 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0643c6f3-b637-4ea4-982d-08af86ce1ff7" containerName="cleanup" Jul 24 01:00:25.342900 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342810 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f65f92-4178-44ab-8aea-efa9de9e4a46" containerName="cleanup" Jul 24 01:00:25.342900 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342821 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="09982e89-a288-45b7-8e99-f6fb8c0a79b2" containerName="cleanup" Jul 24 01:00:25.342900 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.342827 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c672b2-70ad-44ae-99b5-0e273c42908a" containerName="cleanup" Jul 24 01:00:25.346273 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.346248 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:25.349023 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.349005 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 01:00:25.353505 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.353481 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf"] Jul 24 01:00:25.442151 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.442116 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6933996-23d3-4246-a5b8-c6dc571889b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf\" (UID: \"e6933996-23d3-4246-a5b8-c6dc571889b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:25.543528 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.543491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6933996-23d3-4246-a5b8-c6dc571889b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf\" (UID: \"e6933996-23d3-4246-a5b8-c6dc571889b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:25.544162 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.544143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6933996-23d3-4246-a5b8-c6dc571889b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf\" (UID: \"e6933996-23d3-4246-a5b8-c6dc571889b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:25.659929 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.659813 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:25.820548 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.820505 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf"] Jul 24 01:00:25.863512 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:25.863482 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" event={"ID":"e6933996-23d3-4246-a5b8-c6dc571889b7","Type":"ContainerStarted","Data":"c0e9e70265e4557d9d9931cbb7aa971dc3f5739818eaac971930f2190a1a2454"} Jul 24 01:00:26.871121 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:26.871086 2585 generic.go:356] "Generic (PLEG): container finished" podID="e6933996-23d3-4246-a5b8-c6dc571889b7" containerID="e6af85cda2a706d6f1bb3b21e34971cf1a78a5993eadb5cb0681be2954d19bcc" exitCode=0 Jul 24 01:00:26.871527 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:26.871176 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" event={"ID":"e6933996-23d3-4246-a5b8-c6dc571889b7","Type":"ContainerDied","Data":"e6af85cda2a706d6f1bb3b21e34971cf1a78a5993eadb5cb0681be2954d19bcc"} Jul 24 01:00:28.017827 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.017798 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:28.067920 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.067893 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6933996-23d3-4246-a5b8-c6dc571889b7-maas-api-service-ca\") pod \"e6933996-23d3-4246-a5b8-c6dc571889b7\" (UID: \"e6933996-23d3-4246-a5b8-c6dc571889b7\") " Jul 24 01:00:28.068253 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.068231 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6933996-23d3-4246-a5b8-c6dc571889b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6933996-23d3-4246-a5b8-c6dc571889b7" (UID: "e6933996-23d3-4246-a5b8-c6dc571889b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:00:28.169396 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.169318 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6933996-23d3-4246-a5b8-c6dc571889b7-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:00:28.886169 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.886132 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" event={"ID":"e6933996-23d3-4246-a5b8-c6dc571889b7","Type":"ContainerDied","Data":"c0e9e70265e4557d9d9931cbb7aa971dc3f5739818eaac971930f2190a1a2454"} Jul 24 01:00:28.886169 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.886163 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf" Jul 24 01:00:28.886381 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:28.886183 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0e9e70265e4557d9d9931cbb7aa971dc3f5739818eaac971930f2190a1a2454" Jul 24 01:00:35.831906 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:35.831853 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq"] Jul 24 01:00:35.832330 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:35.832122 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" podUID="78da9f13-4e75-432e-ae55-85f20769d8d2" containerName="maas-api" containerID="cri-o://2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113" gracePeriod=30 Jul 24 01:00:35.891260 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:35.891219 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss"] Jul 24 01:00:35.893712 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:35.893692 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-4fa1d5-29747580-ndmss"] Jul 24 01:00:36.115229 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.115195 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 01:00:36.244233 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.244207 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78da9f13-4e75-432e-ae55-85f20769d8d2-openshift-service-ca\") pod \"78da9f13-4e75-432e-ae55-85f20769d8d2\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " Jul 24 01:00:36.244411 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.244260 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls\") pod \"78da9f13-4e75-432e-ae55-85f20769d8d2\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " Jul 24 01:00:36.244411 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.244285 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9sjf\" (UniqueName: \"kubernetes.io/projected/78da9f13-4e75-432e-ae55-85f20769d8d2-kube-api-access-h9sjf\") pod \"78da9f13-4e75-432e-ae55-85f20769d8d2\" (UID: \"78da9f13-4e75-432e-ae55-85f20769d8d2\") " Jul 24 01:00:36.244640 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.244616 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78da9f13-4e75-432e-ae55-85f20769d8d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78da9f13-4e75-432e-ae55-85f20769d8d2" (UID: "78da9f13-4e75-432e-ae55-85f20769d8d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:00:36.246419 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.246392 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78da9f13-4e75-432e-ae55-85f20769d8d2" (UID: "78da9f13-4e75-432e-ae55-85f20769d8d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 01:00:36.246532 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.246479 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78da9f13-4e75-432e-ae55-85f20769d8d2-kube-api-access-h9sjf" (OuterVolumeSpecName: "kube-api-access-h9sjf") pod "78da9f13-4e75-432e-ae55-85f20769d8d2" (UID: "78da9f13-4e75-432e-ae55-85f20769d8d2"). InnerVolumeSpecName "kube-api-access-h9sjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:00:36.345657 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.345574 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78da9f13-4e75-432e-ae55-85f20769d8d2-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:00:36.345657 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.345605 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78da9f13-4e75-432e-ae55-85f20769d8d2-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:00:36.345657 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.345614 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9sjf\" (UniqueName: \"kubernetes.io/projected/78da9f13-4e75-432e-ae55-85f20769d8d2-kube-api-access-h9sjf\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:00:36.934846 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.934808 2585 generic.go:356] "Generic (PLEG): container finished" podID="78da9f13-4e75-432e-ae55-85f20769d8d2" containerID="2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113" exitCode=0 Jul 24 01:00:36.935324 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.934877 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" Jul 24 01:00:36.935324 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.934901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" event={"ID":"78da9f13-4e75-432e-ae55-85f20769d8d2","Type":"ContainerDied","Data":"2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113"} Jul 24 01:00:36.935324 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.934968 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq" event={"ID":"78da9f13-4e75-432e-ae55-85f20769d8d2","Type":"ContainerDied","Data":"5b4014abb7e8780d9640032aa758d4cafafc3afee5117e77d7dd7091fff03b9a"} Jul 24 01:00:36.935324 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.935003 2585 scope.go:117] "RemoveContainer" containerID="2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113" Jul 24 01:00:36.945001 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.944983 2585 scope.go:117] "RemoveContainer" containerID="2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113" Jul 24 01:00:36.945268 ip-10-0-136-102 kubenswrapper[2585]: E0724 01:00:36.945249 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113\": container with ID starting with 2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113 not found: ID does not exist" containerID="2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113" Jul 24 01:00:36.945328 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.945278 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113"} err="failed to get container status \"2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113\": rpc error: code = NotFound desc = could not find container \"2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113\": container with ID starting with 2630bf908c6ae5594462b50b6c6c08fb501a5a26dee7a7387af4bb10c6a29113 not found: ID does not exist" Jul 24 01:00:36.960370 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.960346 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq"] Jul 24 01:00:36.962941 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:36.962919 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fa1d5-7f87b94f58-hbphq"] Jul 24 01:00:37.799408 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:37.799375 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0643c6f3-b637-4ea4-982d-08af86ce1ff7" path="/var/lib/kubelet/pods/0643c6f3-b637-4ea4-982d-08af86ce1ff7/volumes" Jul 24 01:00:37.799750 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:37.799737 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78da9f13-4e75-432e-ae55-85f20769d8d2" path="/var/lib/kubelet/pods/78da9f13-4e75-432e-ae55-85f20769d8d2/volumes" Jul 24 01:00:39.137381 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:39.137350 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 01:00:40.419396 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:40.419358 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf"] Jul 24 01:00:40.421851 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:40.421828 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4fa1d5-wv7zf"] Jul 24 01:00:41.799473 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:41.799432 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6933996-23d3-4246-a5b8-c6dc571889b7" path="/var/lib/kubelet/pods/e6933996-23d3-4246-a5b8-c6dc571889b7/volumes" Jul 24 01:00:47.514361 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514321 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc"] Jul 24 01:00:47.514764 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514750 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78da9f13-4e75-432e-ae55-85f20769d8d2" containerName="maas-api" Jul 24 01:00:47.514764 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514765 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="78da9f13-4e75-432e-ae55-85f20769d8d2" containerName="maas-api" Jul 24 01:00:47.514856 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514787 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6933996-23d3-4246-a5b8-c6dc571889b7" containerName="revoke-keys" Jul 24 01:00:47.514856 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514793 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6933996-23d3-4246-a5b8-c6dc571889b7" containerName="revoke-keys" Jul 24 01:00:47.514856 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514873 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6933996-23d3-4246-a5b8-c6dc571889b7" containerName="revoke-keys" Jul 24 01:00:47.514856 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.514882 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="78da9f13-4e75-432e-ae55-85f20769d8d2" containerName="maas-api" Jul 24 01:00:47.518668 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.518650 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:47.521886 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.521835 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-865ss\"" Jul 24 01:00:47.526991 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.526958 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc"] Jul 24 01:00:47.649434 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.649401 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b11830da-99d4-46db-afd3-9e6d390d2844-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc\" (UID: \"b11830da-99d4-46db-afd3-9e6d390d2844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:47.750698 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.750666 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b11830da-99d4-46db-afd3-9e6d390d2844-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc\" (UID: \"b11830da-99d4-46db-afd3-9e6d390d2844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:47.751317 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.751299 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b11830da-99d4-46db-afd3-9e6d390d2844-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc\" (UID: \"b11830da-99d4-46db-afd3-9e6d390d2844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:47.839415 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.839390 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:47.980197 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:47.980165 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc"] Jul 24 01:00:47.983555 ip-10-0-136-102 kubenswrapper[2585]: W0724 01:00:47.983520 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11830da_99d4_46db_afd3_9e6d390d2844.slice/crio-910d966e09bc733f0e33c1fa24c08bb19858ccb136acbb29ba3a8ac8bba1ce80 WatchSource:0}: Error finding container 910d966e09bc733f0e33c1fa24c08bb19858ccb136acbb29ba3a8ac8bba1ce80: Status 404 returned error can't find the container with id 910d966e09bc733f0e33c1fa24c08bb19858ccb136acbb29ba3a8ac8bba1ce80 Jul 24 01:00:48.006436 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:48.006396 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" event={"ID":"b11830da-99d4-46db-afd3-9e6d390d2844","Type":"ContainerStarted","Data":"910d966e09bc733f0e33c1fa24c08bb19858ccb136acbb29ba3a8ac8bba1ce80"} Jul 24 01:00:49.014727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:49.014685 2585 generic.go:356] "Generic (PLEG): container finished" podID="b11830da-99d4-46db-afd3-9e6d390d2844" containerID="d2e660f2f96e552aa86b979166b190bcbda68872530888bd8962d31a66ba49ae" exitCode=0 Jul 24 01:00:49.014727 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:49.014714 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" event={"ID":"b11830da-99d4-46db-afd3-9e6d390d2844","Type":"ContainerDied","Data":"d2e660f2f96e552aa86b979166b190bcbda68872530888bd8962d31a66ba49ae"} Jul 24 01:00:50.160819 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:50.160796 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:50.276147 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:50.276066 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b11830da-99d4-46db-afd3-9e6d390d2844-maas-api-service-ca\") pod \"b11830da-99d4-46db-afd3-9e6d390d2844\" (UID: \"b11830da-99d4-46db-afd3-9e6d390d2844\") " Jul 24 01:00:50.276492 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:50.276465 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11830da-99d4-46db-afd3-9e6d390d2844-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b11830da-99d4-46db-afd3-9e6d390d2844" (UID: "b11830da-99d4-46db-afd3-9e6d390d2844"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:00:50.377234 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:50.377182 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b11830da-99d4-46db-afd3-9e6d390d2844-maas-api-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:00:51.028288 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:51.028259 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" Jul 24 01:00:51.028288 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:51.028269 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc" event={"ID":"b11830da-99d4-46db-afd3-9e6d390d2844","Type":"ContainerDied","Data":"910d966e09bc733f0e33c1fa24c08bb19858ccb136acbb29ba3a8ac8bba1ce80"} Jul 24 01:00:51.028525 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:51.028318 2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="910d966e09bc733f0e33c1fa24c08bb19858ccb136acbb29ba3a8ac8bba1ce80" Jul 24 01:00:58.066019 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:58.065982 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb"] Jul 24 01:00:58.069253 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:58.069225 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-44952a-29747580-4vhxb"] Jul 24 01:00:59.799494 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:00:59.799461 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c672b2-70ad-44ae-99b5-0e273c42908a" path="/var/lib/kubelet/pods/b4c672b2-70ad-44ae-99b5-0e273c42908a/volumes" Jul 24 01:01:01.529079 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:01.529040 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 01:01:02.579590 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:02.579555 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc"] Jul 24 01:01:02.582123 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:02.582095 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-44952a-v2bmc"] Jul 24 01:01:03.800462 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:03.800427 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11830da-99d4-46db-afd3-9e6d390d2844" path="/var/lib/kubelet/pods/b11830da-99d4-46db-afd3-9e6d390d2844/volumes" Jul 24 01:01:19.736326 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:19.736121 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t"] Jul 24 01:01:19.737411 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:19.736377 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" podUID="68847b5e-e810-4fdd-90c9-726bd6db2b78" containerName="maas-api" containerID="cri-o://0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215" gracePeriod=30 Jul 24 01:01:19.761026 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:19.760991 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp"] Jul 24 01:01:19.766975 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:19.766953 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-91a75b-29747580-gd8mp"] Jul 24 01:01:19.803038 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:19.802990 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09982e89-a288-45b7-8e99-f6fb8c0a79b2" path="/var/lib/kubelet/pods/09982e89-a288-45b7-8e99-f6fb8c0a79b2/volumes" Jul 24 01:01:20.006389 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.006363 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 01:01:20.132324 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.132280 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzdhz\" (UniqueName: \"kubernetes.io/projected/68847b5e-e810-4fdd-90c9-726bd6db2b78-kube-api-access-dzdhz\") pod \"68847b5e-e810-4fdd-90c9-726bd6db2b78\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " Jul 24 01:01:20.132495 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.132364 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68847b5e-e810-4fdd-90c9-726bd6db2b78-openshift-service-ca\") pod \"68847b5e-e810-4fdd-90c9-726bd6db2b78\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " Jul 24 01:01:20.132495 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.132479 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68847b5e-e810-4fdd-90c9-726bd6db2b78-maas-api-tls\") pod \"68847b5e-e810-4fdd-90c9-726bd6db2b78\" (UID: \"68847b5e-e810-4fdd-90c9-726bd6db2b78\") " Jul 24 01:01:20.132644 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.132627 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68847b5e-e810-4fdd-90c9-726bd6db2b78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68847b5e-e810-4fdd-90c9-726bd6db2b78" (UID: "68847b5e-e810-4fdd-90c9-726bd6db2b78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 01:01:20.132829 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.132812 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68847b5e-e810-4fdd-90c9-726bd6db2b78-openshift-service-ca\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:01:20.134458 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.134425 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68847b5e-e810-4fdd-90c9-726bd6db2b78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68847b5e-e810-4fdd-90c9-726bd6db2b78" (UID: "68847b5e-e810-4fdd-90c9-726bd6db2b78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 01:01:20.134587 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.134490 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68847b5e-e810-4fdd-90c9-726bd6db2b78-kube-api-access-dzdhz" (OuterVolumeSpecName: "kube-api-access-dzdhz") pod "68847b5e-e810-4fdd-90c9-726bd6db2b78" (UID: "68847b5e-e810-4fdd-90c9-726bd6db2b78"). InnerVolumeSpecName "kube-api-access-dzdhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 01:01:20.211423 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.211386 2585 generic.go:356] "Generic (PLEG): container finished" podID="68847b5e-e810-4fdd-90c9-726bd6db2b78" containerID="0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215" exitCode=0 Jul 24 01:01:20.211423 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.211426 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" Jul 24 01:01:20.211613 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.211468 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" event={"ID":"68847b5e-e810-4fdd-90c9-726bd6db2b78","Type":"ContainerDied","Data":"0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215"} Jul 24 01:01:20.211613 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.211543 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t" event={"ID":"68847b5e-e810-4fdd-90c9-726bd6db2b78","Type":"ContainerDied","Data":"4fecd95db30fb03fcf6f2d2f1494141715533e12b7c620fc8edfa7c4f21a9f80"} Jul 24 01:01:20.211613 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.211569 2585 scope.go:117] "RemoveContainer" containerID="0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215" Jul 24 01:01:20.221698 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.221682 2585 scope.go:117] "RemoveContainer" containerID="0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215" Jul 24 01:01:20.221996 ip-10-0-136-102 kubenswrapper[2585]: E0724 01:01:20.221977 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215\": container with ID starting with 0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215 not found: ID does not exist" containerID="0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215" Jul 24 01:01:20.222064 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.222004 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215"} err="failed to get container status \"0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215\": rpc error: code = NotFound desc = could not find container \"0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215\": container with ID starting with 0cc9aaf37fe947aff801b7571b7d26668fa2eaba3afe7049cf041916bc315215 not found: ID does not exist" Jul 24 01:01:20.234186 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.234163 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdhz\" (UniqueName: \"kubernetes.io/projected/68847b5e-e810-4fdd-90c9-726bd6db2b78-kube-api-access-dzdhz\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:01:20.234275 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.234188 2585 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68847b5e-e810-4fdd-90c9-726bd6db2b78-maas-api-tls\") on node \"ip-10-0-136-102.ec2.internal\" DevicePath \"\"" Jul 24 01:01:20.234654 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.234636 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t"] Jul 24 01:01:20.238194 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:20.238175 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-91a75b-595dcd4bc8-54r7t"] Jul 24 01:01:21.799351 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:21.799320 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68847b5e-e810-4fdd-90c9-726bd6db2b78" path="/var/lib/kubelet/pods/68847b5e-e810-4fdd-90c9-726bd6db2b78/volumes" Jul 24 01:01:22.731617 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:22.731584 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-589ln"] Jul 24 01:01:37.120333 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.120300 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kwnvg_9e0454cd-f302-4c17-a44a-1ab0c3afd4ed/manager/0.log" Jul 24 01:01:37.235643 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.235604 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tjs92_68d247fa-5dfc-4592-b64e-21791ab80e24/manager/0.log" Jul 24 01:01:37.359337 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.359307 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9bbd68d9-nwxlt_d7a46795-810e-4942-9f0a-7b761cefd871/manager/0.log" Jul 24 01:01:37.491433 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.491362 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kcmk5_3b61fb86-63ef-44d6-87c6-27376f9c1a68/server/0.log" Jul 24 01:01:37.613011 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.612981 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-969d5_93a9e45d-b7d4-4c7c-956f-2893d8727066/manager/0.log" Jul 24 01:01:37.726920 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.726895 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-26nfc_71b9799e-d866-49ce-93cc-dc513adae89c/copy-manifests/0.log" Jul 24 01:01:37.738690 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:37.738652 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-26nfc_71b9799e-d866-49ce-93cc-dc513adae89c/manager/0.log" Jul 24 01:01:38.353194 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:38.353163 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-dc745548d-6gmb9_c2544ac4-8cbc-451a-9fcf-5317fd95bb03/maas-api/0.log" Jul 24 01:01:38.702783 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:38.702702 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747580-dw95s_a4f65f92-4178-44ab-8aea-efa9de9e4a46/cleanup/0.log" Jul 24 01:01:38.831289 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:38.831263 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8mj77_6ff37e38-0287-439f-9859-b989e4613269/postgres/0.log" Jul 24 01:01:39.659089 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.659055 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c_27793db7-7125-4f99-b2d6-e6f0a4cdc4ab/util/0.log" Jul 24 01:01:39.664795 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.664771 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c_27793db7-7125-4f99-b2d6-e6f0a4cdc4ab/pull/0.log" Jul 24 01:01:39.670002 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.669983 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c_27793db7-7125-4f99-b2d6-e6f0a4cdc4ab/extract/0.log" Jul 24 01:01:39.787292 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.787244 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt_2088bf82-d4fd-4391-a56f-22fd4539fb81/pull/0.log" Jul 24 01:01:39.796730 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.796707 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt_2088bf82-d4fd-4391-a56f-22fd4539fb81/extract/0.log" Jul 24 01:01:39.802810 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.802795 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt_2088bf82-d4fd-4391-a56f-22fd4539fb81/util/0.log" Jul 24 01:01:39.918723 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.918623 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx_98217b70-405b-44f1-98aa-0772e459ff68/util/0.log" Jul 24 01:01:39.924718 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.924693 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx_98217b70-405b-44f1-98aa-0772e459ff68/pull/0.log" Jul 24 01:01:39.930532 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:39.930479 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx_98217b70-405b-44f1-98aa-0772e459ff68/extract/0.log" Jul 24 01:01:40.046144 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.046111 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68_feb330f7-a602-4a9c-8b62-031ac0c65c53/util/0.log" Jul 24 01:01:40.051944 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.051924 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68_feb330f7-a602-4a9c-8b62-031ac0c65c53/pull/0.log" Jul 24 01:01:40.057784 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.057767 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68_feb330f7-a602-4a9c-8b62-031ac0c65c53/extract/0.log" Jul 24 01:01:40.346686 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.346659 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-656c6_1324ba49-6b60-407e-85f9-c1ed0d01c917/manager/0.log" Jul 24 01:01:40.464348 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.464312 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-th6ct_210accd6-b966-48bf-b346-f35109aaa8f9/manager/0.log" Jul 24 01:01:40.579505 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.579481 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z59hk_3d64bb08-9653-494c-8f76-f1f9c9fee2c7/kuadrant-console-plugin/0.log" Jul 24 01:01:40.702292 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.702207 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fgfqk_fdc3ae66-934f-428d-8916-3ad1a64463db/registry-server/0.log" Jul 24 01:01:40.821183 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.821146 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-95rwn_e3f3a182-f0ff-4c0f-bf17-2e8e458f7808/manager/0.log" Jul 24 01:01:40.933521 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:40.933486 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-589ln_3c81af3b-ef11-4850-9d1a-39911fe34aba/limitador/0.log" Jul 24 01:01:41.652005 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:41.651972 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-f5sh5_070900fd-ba4a-4155-9f55-5d2a6c76ece0/kube-auth-proxy/0.log" Jul 24 01:01:42.135209 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:42.135182 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x24f_262c57a9-2608-4235-bb94-b67ac50ac889/payload-processing/0.log" Jul 24 01:01:42.250073 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:42.250045 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-654cb6b4b5-ps8tt_5731834c-6e4d-474c-91d3-e92d41b96d00/router/0.log" Jul 24 01:01:42.597370 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:42.597340 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gzj75_3bcc91f9-06c7-490c-aa7c-43c5c2f11aaf/main/0.log" Jul 24 01:01:42.943629 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:42.943551 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2n95h_0ce82edb-f2ca-4760-949f-0ba9d9674c55/main/0.log" Jul 24 01:01:54.161269 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:54.161236 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-djmtj_3632857f-077a-4c1e-b064-dcb861c45095/global-pull-secret-syncer/0.log" Jul 24 01:01:54.329829 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:54.329796 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lzc8x_8cdbe738-d480-4b22-ad50-a0a84c52db0a/konnectivity-agent/0.log" Jul 24 01:01:54.393127 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:54.393105 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-102.ec2.internal_76d1c0db1c0eb2e7cc88fd5e78b49f45/haproxy/0.log" Jul 24 01:01:58.780194 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.780164 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c_27793db7-7125-4f99-b2d6-e6f0a4cdc4ab/extract/0.log" Jul 24 01:01:58.806955 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.806927 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c_27793db7-7125-4f99-b2d6-e6f0a4cdc4ab/util/0.log" Jul 24 01:01:58.832793 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.832726 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bdr8c_27793db7-7125-4f99-b2d6-e6f0a4cdc4ab/pull/0.log" Jul 24 01:01:58.863926 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.863895 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt_2088bf82-d4fd-4391-a56f-22fd4539fb81/extract/0.log" Jul 24 01:01:58.883750 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.883719 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt_2088bf82-d4fd-4391-a56f-22fd4539fb81/util/0.log" Jul 24 01:01:58.903390 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.903364 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0cwfjt_2088bf82-d4fd-4391-a56f-22fd4539fb81/pull/0.log" Jul 24 01:01:58.937644 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.937620 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx_98217b70-405b-44f1-98aa-0772e459ff68/extract/0.log" Jul 24 01:01:58.959835 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.959811 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx_98217b70-405b-44f1-98aa-0772e459ff68/util/0.log" Jul 24 01:01:58.987104 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:58.987077 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7344vpx_98217b70-405b-44f1-98aa-0772e459ff68/pull/0.log" Jul 24 01:01:59.012316 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.012297 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68_feb330f7-a602-4a9c-8b62-031ac0c65c53/extract/0.log" Jul 24 01:01:59.031850 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.031825 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68_feb330f7-a602-4a9c-8b62-031ac0c65c53/util/0.log" Jul 24 01:01:59.053166 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.053145 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hnn68_feb330f7-a602-4a9c-8b62-031ac0c65c53/pull/0.log" Jul 24 01:01:59.593136 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.593108 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-656c6_1324ba49-6b60-407e-85f9-c1ed0d01c917/manager/0.log" Jul 24 01:01:59.618691 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.618668 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-th6ct_210accd6-b966-48bf-b346-f35109aaa8f9/manager/0.log" Jul 24 01:01:59.650027 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.650004 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-z59hk_3d64bb08-9653-494c-8f76-f1f9c9fee2c7/kuadrant-console-plugin/0.log" Jul 24 01:01:59.690354 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.690331 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fgfqk_fdc3ae66-934f-428d-8916-3ad1a64463db/registry-server/0.log" Jul 24 01:01:59.841847 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.841817 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-95rwn_e3f3a182-f0ff-4c0f-bf17-2e8e458f7808/manager/0.log" Jul 24 01:01:59.867462 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:01:59.867388 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-589ln_3c81af3b-ef11-4850-9d1a-39911fe34aba/limitador/0.log" Jul 24 01:02:01.776028 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:01.775995 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-bdrx8_6d8e7797-9ab7-4b76-a5cf-f6b16f548776/cluster-monitoring-operator/0.log" Jul 24 01:02:02.041593 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:02.041512 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vlnq_568bf52a-5f92-44ca-b3d4-278b6ae3d209/node-exporter/0.log" Jul 24 01:02:02.066833 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:02.066809 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vlnq_568bf52a-5f92-44ca-b3d4-278b6ae3d209/kube-rbac-proxy/0.log" Jul 24 01:02:02.090368 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:02.090345 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vlnq_568bf52a-5f92-44ca-b3d4-278b6ae3d209/init-textfile/0.log" Jul 24 01:02:02.482939 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:02.482912 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-8g56k_e2d52d30-41dd-450d-8c10-0a4655c2455d/prometheus-operator-admission-webhook/0.log" Jul 24 01:02:03.861112 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:03.861083 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-tgg5c_ca23905c-87a5-4884-ba50-74e081b46cea/networking-console-plugin/0.log" Jul 24 01:02:04.444906 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.444874 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk"] Jul 24 01:02:04.445298 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.445285 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68847b5e-e810-4fdd-90c9-726bd6db2b78" containerName="maas-api" Jul 24 01:02:04.445298 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.445298 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="68847b5e-e810-4fdd-90c9-726bd6db2b78" containerName="maas-api" Jul 24 01:02:04.445429 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.445311 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11830da-99d4-46db-afd3-9e6d390d2844" containerName="revoke-keys" Jul 24 01:02:04.445429 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.445321 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11830da-99d4-46db-afd3-9e6d390d2844" containerName="revoke-keys" Jul 24 01:02:04.445429 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.445427 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11830da-99d4-46db-afd3-9e6d390d2844" containerName="revoke-keys" Jul 24 01:02:04.445558 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.445443 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="68847b5e-e810-4fdd-90c9-726bd6db2b78" containerName="maas-api" Jul 24 01:02:04.448664 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.448649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.451017 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.450988 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v8jb\"/\"kube-root-ca.crt\"" Jul 24 01:02:04.451239 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.451223 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v8jb\"/\"openshift-service-ca.crt\"" Jul 24 01:02:04.452334 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.452297 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7v8jb\"/\"default-dockercfg-hlg9z\"" Jul 24 01:02:04.458650 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.458631 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk"] Jul 24 01:02:04.512322 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.512288 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/2.log" Jul 24 01:02:04.516556 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.516536 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-chwt8_fd52790b-d34a-4c5b-8bf9-b1170f865154/console-operator/3.log" Jul 24 01:02:04.526045 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.526008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-podres\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.526164 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.526074 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-sys\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.526164 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.526110 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-lib-modules\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.526164 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.526128 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-proc\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.526164 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.526142 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvq98\" (UniqueName: \"kubernetes.io/projected/84c8667f-61d6-4541-8065-9dfdf6d03dc1-kube-api-access-qvq98\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627001 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.626961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-podres\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627192 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627040 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-sys\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627192 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627063 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-lib-modules\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627192 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627083 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-proc\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627192 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627100 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvq98\" (UniqueName: \"kubernetes.io/projected/84c8667f-61d6-4541-8065-9dfdf6d03dc1-kube-api-access-qvq98\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627192 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627140 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-podres\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627192 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627169 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-proc\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627435 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627192 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-sys\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.627435 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.627224 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84c8667f-61d6-4541-8065-9dfdf6d03dc1-lib-modules\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.635563 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.635536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvq98\" (UniqueName: \"kubernetes.io/projected/84c8667f-61d6-4541-8065-9dfdf6d03dc1-kube-api-access-qvq98\") pod \"perf-node-gather-daemonset-hvvdk\" (UID: \"84c8667f-61d6-4541-8065-9dfdf6d03dc1\") " pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.758699 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.758623 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:04.897977 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:04.897951 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk"] Jul 24 01:02:04.899973 ip-10-0-136-102 kubenswrapper[2585]: W0724 01:02:04.899939 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84c8667f_61d6_4541_8065_9dfdf6d03dc1.slice/crio-13d2c5b033cda195bf05282425fd9e53c5518d096701f61e41a2fc64687db101 WatchSource:0}: Error finding container 13d2c5b033cda195bf05282425fd9e53c5518d096701f61e41a2fc64687db101: Status 404 returned error can't find the container with id 13d2c5b033cda195bf05282425fd9e53c5518d096701f61e41a2fc64687db101 Jul 24 01:02:05.092511 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:05.092486 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-674849cc4f-jlpnv_32aa6363-aae9-4100-915d-751b74d4beb3/console/0.log" Jul 24 01:02:05.498229 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:05.498143 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" event={"ID":"84c8667f-61d6-4541-8065-9dfdf6d03dc1","Type":"ContainerStarted","Data":"3c26d8abfecbc0c9313e9e469960f1aede51554352e8844e56a1fd45a7b2916b"} Jul 24 01:02:05.498229 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:05.498202 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" event={"ID":"84c8667f-61d6-4541-8065-9dfdf6d03dc1","Type":"ContainerStarted","Data":"13d2c5b033cda195bf05282425fd9e53c5518d096701f61e41a2fc64687db101"} Jul 24 01:02:05.498229 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:05.498226 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:05.516842 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:05.516795 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" podStartSLOduration=1.516779136 podStartE2EDuration="1.516779136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 01:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 01:02:05.513813798 +0000 UTC m=+3102.246448822" watchObservedRunningTime="2026-07-24 01:02:05.516779136 +0000 UTC m=+3102.249414160" Jul 24 01:02:05.693581 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:05.693556 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-8hxz8_8e998951-5d35-4949-b85f-1ebe6b302c3d/volume-data-source-validator/0.log" Jul 24 01:02:06.779635 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:06.779605 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nmfqx_0f4fe5bb-86b1-4155-a8cd-f3a151519316/dns/0.log" Jul 24 01:02:06.802755 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:06.802728 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nmfqx_0f4fe5bb-86b1-4155-a8cd-f3a151519316/kube-rbac-proxy/0.log" Jul 24 01:02:06.945737 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:06.945706 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xjx9t_ee0c4f9c-792d-4b61-9493-44a83f3632ba/dns-node-resolver/0.log" Jul 24 01:02:07.632661 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:07.632632 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nvdwf_1d88949d-b7a2-4105-9d66-793b76308f2b/node-ca/0.log" Jul 24 01:02:08.734353 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:08.734324 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-f5sh5_070900fd-ba4a-4155-9f55-5d2a6c76ece0/kube-auth-proxy/0.log" Jul 24 01:02:08.896577 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:08.896547 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x24f_262c57a9-2608-4235-bb94-b67ac50ac889/payload-processing/0.log" Jul 24 01:02:08.921487 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:08.921456 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-654cb6b4b5-ps8tt_5731834c-6e4d-474c-91d3-e92d41b96d00/router/0.log" Jul 24 01:02:09.718594 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:09.718571 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jh5zk_7d0f4562-f05a-4a3c-9e4b-c45eda7ffaee/serve-healthcheck-canary/0.log" Jul 24 01:02:10.224796 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:10.224763 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-qffj9_0bb576d5-b2da-4ff1-9ca6-68f74da0ab02/insights-operator/0.log" Jul 24 01:02:10.225536 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:10.225517 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-qffj9_0bb576d5-b2da-4ff1-9ca6-68f74da0ab02/insights-operator/1.log" Jul 24 01:02:10.394106 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:10.394082 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p9zgc_24722498-2095-48cc-941a-c82ef22e9ff1/kube-rbac-proxy/0.log" Jul 24 01:02:10.414201 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:10.414174 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p9zgc_24722498-2095-48cc-941a-c82ef22e9ff1/exporter/0.log" Jul 24 01:02:10.435644 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:10.435623 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p9zgc_24722498-2095-48cc-941a-c82ef22e9ff1/extractor/0.log" Jul 24 01:02:11.514991 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:11.514967 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7v8jb/perf-node-gather-daemonset-hvvdk" Jul 24 01:02:12.561167 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:12.561140 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kwnvg_9e0454cd-f302-4c17-a44a-1ab0c3afd4ed/manager/0.log" Jul 24 01:02:12.751174 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:12.751142 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tjs92_68d247fa-5dfc-4592-b64e-21791ab80e24/manager/0.log" Jul 24 01:02:12.873841 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:12.873758 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9bbd68d9-nwxlt_d7a46795-810e-4942-9f0a-7b761cefd871/manager/0.log" Jul 24 01:02:12.896276 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:12.896251 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-kcmk5_3b61fb86-63ef-44d6-87c6-27376f9c1a68/server/0.log" Jul 24 01:02:12.968637 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:12.968609 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-969d5_93a9e45d-b7d4-4c7c-956f-2893d8727066/manager/0.log" Jul 24 01:02:13.056794 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:13.056765 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-26nfc_71b9799e-d866-49ce-93cc-dc513adae89c/manager/0.log" Jul 24 01:02:13.077665 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:13.077638 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7cdb55f47c-26nfc_71b9799e-d866-49ce-93cc-dc513adae89c/copy-manifests/0.log" Jul 24 01:02:14.952134 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:14.952053 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-f5sh5_070900fd-ba4a-4155-9f55-5d2a6c76ece0/kube-auth-proxy/0.log" Jul 24 01:02:15.069702 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:15.069669 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x24f_262c57a9-2608-4235-bb94-b67ac50ac889/payload-processing/0.log" Jul 24 01:02:15.084430 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:15.084404 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-654cb6b4b5-ps8tt_5731834c-6e4d-474c-91d3-e92d41b96d00/router/0.log" Jul 24 01:02:16.705194 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:16.705168 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-57dc97f664-f5sh5_070900fd-ba4a-4155-9f55-5d2a6c76ece0/kube-auth-proxy/0.log" Jul 24 01:02:16.844161 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:16.844130 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-5x24f_262c57a9-2608-4235-bb94-b67ac50ac889/payload-processing/0.log" Jul 24 01:02:16.865339 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:16.865314 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-654cb6b4b5-ps8tt_5731834c-6e4d-474c-91d3-e92d41b96d00/router/0.log" Jul 24 01:02:19.745711 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:19.745679 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zn62p_305fa5cc-79a6-4f1f-8d86-39bbf6633f1b/migrator/0.log" Jul 24 01:02:19.766451 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:19.766428 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zn62p_305fa5cc-79a6-4f1f-8d86-39bbf6633f1b/graceful-termination/0.log" Jul 24 01:02:20.118854 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:20.118820 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw_230ddb24-8a95-4058-b34a-25b6d01076d1/kube-storage-version-migrator-operator/1.log" Jul 24 01:02:20.119738 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:20.119708 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-gnhlw_230ddb24-8a95-4058-b34a-25b6d01076d1/kube-storage-version-migrator-operator/0.log" Jul 24 01:02:21.222781 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.222750 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-878m4_0c1b0e6d-0613-4706-84a1-795bbef96de4/kube-multus/0.log" Jul 24 01:02:21.400595 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.400563 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/kube-multus-additional-cni-plugins/0.log" Jul 24 01:02:21.421281 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.421256 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/egress-router-binary-copy/0.log" Jul 24 01:02:21.441934 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.441909 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/cni-plugins/0.log" Jul 24 01:02:21.464725 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.464703 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/bond-cni-plugin/0.log" Jul 24 01:02:21.485118 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.485055 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/routeoverride-cni/0.log" Jul 24 01:02:21.506393 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.506367 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/whereabouts-cni-bincopy/0.log" Jul 24 01:02:21.525368 ip-10-0-136-102 kubenswrapper[2585]: I0724 01:02:21.525343 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tdltg_acb33309-ccf8-4df1-a74a-2f53fc79bee3/whereabouts-cni/0.log"