Jul 24 13:36:22.503622 ip-10-0-138-248 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 13:36:22.503633 ip-10-0-138-248 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 13:36:22.503642 ip-10-0-138-248 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 13:36:22.503980 ip-10-0-138-248 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 13:36:32.589527 ip-10-0-138-248 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 13:36:32.589547 ip-10-0-138-248 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3d301a6be4684f6b9fdfc11f35b483d0 -- Jul 24 13:38:21.946005 ip-10-0-138-248 systemd[1]: Starting Kubernetes Kubelet... Jul 24 13:38:22.324950 ip-10-0-138-248 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 13:38:22.324950 ip-10-0-138-248 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 13:38:22.324950 ip-10-0-138-248 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 13:38:22.324950 ip-10-0-138-248 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 13:38:22.324950 ip-10-0-138-248 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 13:38:22.325845 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.325773 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 13:38:22.332777 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332740 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:22.332777 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332772 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:22.332777 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332777 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:22.332777 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332780 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:22.332777 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332783 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332786 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332789 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332792 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332795 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332797 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332800 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332802 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332805 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332807 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332810 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332812 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332815 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332823 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332826 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332828 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332831 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332834 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332836 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332839 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:22.332965 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332849 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332852 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332854 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332857 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332860 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332862 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332865 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332868 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332871 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332873 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332876 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332878 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332881 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332883 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332887 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332894 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332897 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332900 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332902 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332905 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:22.333444 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332908 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332910 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332916 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332920 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332923 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332926 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332929 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332932 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332936 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332939 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332942 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332945 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332948 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332950 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332953 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332956 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332958 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332961 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332963 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:22.333957 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332966 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332970 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332974 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332976 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332979 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332982 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332985 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332988 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332991 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332993 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332996 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.332999 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333001 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333004 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333006 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333009 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333012 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333014 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333017 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:22.334408 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333019 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333022 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333025 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333028 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333030 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333033 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333035 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333038 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333040 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333043 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333045 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333048 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333050 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333053 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333055 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333058 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333060 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333741 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333766 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333770 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:22.334911 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333774 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333777 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333780 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333783 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333787 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333791 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333794 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333797 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333799 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333802 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333804 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333807 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333809 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333812 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333816 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333819 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333821 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333824 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333827 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333829 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:22.335384 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333832 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333835 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333837 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333840 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333842 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333845 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333847 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333850 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333853 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333856 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333858 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333861 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333865 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333868 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333872 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333875 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333879 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333881 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333884 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:22.335892 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333887 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333890 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333893 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333896 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333899 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333901 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333904 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333907 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333909 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333912 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333915 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333917 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333920 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333923 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333925 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333928 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333930 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333933 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333935 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333938 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:22.336352 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333940 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333943 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333945 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333948 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333950 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333954 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333957 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333960 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333964 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333966 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333969 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333972 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333974 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333977 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333979 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333982 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333984 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333987 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333989 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333992 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:22.336860 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333994 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.333999 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334002 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334004 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334007 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334010 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334012 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334015 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334017 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334020 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334022 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334025 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334027 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334029 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334032 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334034 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334037 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334402 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334407 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334410 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:22.337343 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334413 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334416 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334418 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334421 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334424 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334427 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334429 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334432 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334434 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334437 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334439 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334442 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334444 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334452 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334455 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334459 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334463 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334466 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334469 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:22.337849 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334471 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334474 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334477 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334479 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334482 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334485 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334487 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334490 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334492 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334495 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334499 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334501 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334504 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334507 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334510 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334512 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334515 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334517 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334520 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334522 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:22.338312 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334525 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334527 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334530 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334532 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334535 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334537 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334540 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334548 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334551 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334554 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334556 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334559 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334561 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334564 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334566 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334569 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334572 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334576 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334579 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334581 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:22.338811 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334584 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334587 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334589 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334592 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334594 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334597 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334599 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334602 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334605 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334608 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334611 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334614 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334616 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334619 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334622 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334624 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334626 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334629 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334632 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:22.339295 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334635 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334639 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334643 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334646 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334649 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334652 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334655 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334659 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334662 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334665 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334670 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334672 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334675 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334678 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334680 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334683 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334686 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.334688 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334766 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334773 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334779 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 13:38:22.339797 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334784 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334787 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334790 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334794 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334799 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334802 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334805 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334808 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334811 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334815 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334820 2568 flags.go:64] FLAG: --cgroup-root="" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334822 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334825 2568 flags.go:64] FLAG: --client-ca-file="" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334829 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334832 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334837 2568 flags.go:64] FLAG: --cluster-domain="" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334839 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334842 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334845 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334849 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334856 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334861 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334864 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 13:38:22.340309 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334867 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334870 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334873 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334876 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334879 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334882 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334886 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334889 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334892 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334894 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334897 2568 flags.go:64] FLAG: --enable-server="true" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334900 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334904 2568 flags.go:64] FLAG: --event-burst="100" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334907 2568 flags.go:64] FLAG: --event-qps="50" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334910 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334913 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334916 2568 flags.go:64] FLAG: --eviction-hard="" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334920 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334922 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334927 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334930 2568 flags.go:64] FLAG: --eviction-soft="" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334933 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334935 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334939 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334942 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 13:38:22.340875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334944 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334947 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334950 2568 flags.go:64] FLAG: --feature-gates="" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334954 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334957 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334961 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334964 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334966 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334969 2568 flags.go:64] FLAG: --help="false" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334972 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334975 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334978 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334981 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334984 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334987 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334990 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334992 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334995 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.334998 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335001 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335004 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335007 2568 flags.go:64] FLAG: --kube-reserved="" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335009 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335012 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 13:38:22.341522 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335015 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335018 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335022 2568 flags.go:64] FLAG: --lock-file="" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335025 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335028 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335030 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335035 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335039 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335042 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335044 2568 flags.go:64] FLAG: --logging-format="text" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335047 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335050 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335053 2568 flags.go:64] FLAG: --manifest-url="" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335056 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335060 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335063 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335067 2568 flags.go:64] FLAG: --max-pods="110" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335070 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335073 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335076 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335078 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335081 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335084 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335087 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335094 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 13:38:22.342126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335097 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335100 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335103 2568 flags.go:64] FLAG: --pod-cidr="" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335105 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335111 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335113 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335117 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335120 2568 flags.go:64] FLAG: --port="10250" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335122 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335126 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-007cbb5172702d080" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335130 2568 flags.go:64] FLAG: --qos-reserved="" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335133 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335135 2568 flags.go:64] FLAG: --register-node="true" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335138 2568 flags.go:64] FLAG: --register-with-taints="" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335142 2568 flags.go:64] FLAG: --registry-burst="10" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335145 2568 flags.go:64] FLAG: --registry-qps="5" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335148 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335151 2568 flags.go:64] FLAG: --reserved-memory="" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335154 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335157 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335160 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335163 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335165 2568 flags.go:64] FLAG: --runonce="false" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335168 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335171 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 13:38:22.342722 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335174 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335177 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335179 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335182 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335185 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335188 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335191 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335194 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335196 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335199 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335202 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335205 2568 flags.go:64] FLAG: --system-cgroups="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335207 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335212 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335215 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335218 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335223 2568 flags.go:64] FLAG: --tls-min-version="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335226 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335228 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335231 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335234 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335237 2568 flags.go:64] FLAG: --v="2" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335241 2568 flags.go:64] FLAG: --version="false" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335245 2568 flags.go:64] FLAG: --vmodule="" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335249 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 13:38:22.343440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.335252 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335359 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335363 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335366 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335369 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335372 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335374 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335377 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335380 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335382 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335385 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335387 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335391 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335393 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335396 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335400 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335404 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335407 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335410 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:22.344081 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335412 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335415 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335418 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335421 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335424 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335427 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335431 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335434 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335436 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335439 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335441 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335444 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335447 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335449 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335452 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335454 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335457 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335459 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335462 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335465 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:22.344539 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335467 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335470 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335472 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335475 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335477 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335480 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335482 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335485 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335487 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335490 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335493 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335495 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335498 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335500 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335503 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335505 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335509 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335512 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335515 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335518 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:22.345043 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335520 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335523 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335525 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335528 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335530 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335533 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335536 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335538 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335541 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335543 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335546 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335550 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335553 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335556 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335558 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335561 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335564 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335566 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335569 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335572 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:22.345534 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335575 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335577 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335580 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335582 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335585 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335587 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335590 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335593 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335597 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335599 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335603 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335606 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335609 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335611 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335614 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335616 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335620 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335624 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335627 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335630 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:22.346029 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.335632 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.336152 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.342451 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.342464 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.342480 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.342485 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342543 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342548 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342551 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342554 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342557 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342560 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342562 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342565 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342568 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:22.346532 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342570 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342573 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342575 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342578 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342581 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342583 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342586 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342588 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342591 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342594 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342596 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342599 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342601 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342604 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342607 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342610 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342612 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342615 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342618 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342621 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:22.346989 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342624 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342626 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342629 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342631 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342634 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342636 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342639 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342641 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342644 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342647 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342649 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342652 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342655 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342657 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342660 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342663 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342665 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342668 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342670 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342673 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:22.347480 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342675 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342678 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342682 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342685 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342688 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342691 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342693 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342696 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342699 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342701 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342704 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342707 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342710 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342713 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342715 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342718 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342721 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342724 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:22.347984 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342727 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342731 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342734 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342737 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342741 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342743 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342763 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342768 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342773 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342777 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342779 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342782 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342785 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342788 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342790 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342793 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342795 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342798 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342800 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:22.348422 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342803 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342806 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342808 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342811 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342814 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342817 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342820 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342822 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342825 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342827 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342830 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342832 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342835 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.342840 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342947 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 13:38:22.348894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342951 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342954 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342957 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342960 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342967 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342970 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342974 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342978 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342981 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342984 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342987 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342989 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342992 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342995 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.342997 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343000 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343002 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343005 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343007 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343010 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 13:38:22.349254 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343013 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343015 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343018 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343021 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343024 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343026 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343029 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343032 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343035 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343037 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343040 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343042 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343045 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343047 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343050 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343052 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343055 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343057 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343059 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343062 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 13:38:22.349725 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343065 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343067 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343069 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343072 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343074 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343077 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343079 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343082 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343084 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343087 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343089 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343092 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343094 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343097 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343100 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343103 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343106 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343109 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343111 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343114 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 13:38:22.350226 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343116 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343118 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343121 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343123 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343127 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343130 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343134 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343137 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343140 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343143 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343145 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343148 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343150 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343153 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343155 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343158 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343160 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343163 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343165 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 13:38:22.350701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343167 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343170 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343172 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343175 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343177 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343180 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343182 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343185 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343188 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343191 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343194 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343196 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343199 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343201 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343204 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343206 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343209 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343211 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 13:38:22.351170 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:22.343214 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.343218 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.343308 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.345153 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.346427 2568 server.go:1019] "Starting client certificate rotation" Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.346559 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.346604 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 13:38:22.351606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.349232 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 13:38:22.351843 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.351009 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 13:38:22.362146 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.362128 2568 log.go:25] "Validated CRI v1 runtime API" Jul 24 13:38:22.364220 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.364206 2568 log.go:25] "Validated CRI v1 image API" Jul 24 13:38:22.365925 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.365910 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 13:38:22.369865 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.369847 2568 fs.go:135] Filesystem UUIDs: map[22a665ad-d56a-4fc8-a846-3e0db2708d7a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c09cb8d1-e52e-446f-8fda-20d028826fd3:/dev/nvme0n1p4] Jul 24 13:38:22.369911 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.369866 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 13:38:22.375588 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.375455 2568 manager.go:216] Machine: {Timestamp:2026-07-24 13:38:22.373383342 +0000 UTC m=+0.313838053 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101854 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2494bb1fb794628c9b319ddb50eaef SystemUUID:ec2494bb-1fb7-9462-8c9b-319ddb50eaef BootID:3d301a6b-e468-4f6b-9fdf-c11f35b483d0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:71:f3:11:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:71:f3:11:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:2f:35:c9:e8:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 13:38:22.375588 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.375570 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 13:38:22.375736 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.375668 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 13:38:22.376684 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.376663 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 13:38:22.376837 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.376687 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-248.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 13:38:22.377143 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.377130 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 13:38:22.377724 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.377713 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 13:38:22.377775 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.377728 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 13:38:22.377775 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.377735 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 13:38:22.378398 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.378387 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 13:38:22.379090 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.379080 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 13:38:22.380795 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380784 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 24 13:38:22.380900 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380891 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 13:38:22.380941 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380918 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 24 13:38:22.380941 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380927 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 13:38:22.380941 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380939 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 13:38:22.381040 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380949 2568 kubelet.go:398] "Adding apiserver pod source" Jul 24 13:38:22.381040 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.380957 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 13:38:22.381300 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.381289 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 13:38:22.381332 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.381310 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 13:38:22.383423 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.383407 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 13:38:22.385101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.385087 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 13:38:22.385174 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.385117 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 13:38:22.386513 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386501 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386517 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386523 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386529 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386537 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386543 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386551 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 13:38:22.386556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386556 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 13:38:22.386774 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386562 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 13:38:22.386774 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386567 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 13:38:22.386774 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386576 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 13:38:22.386774 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.386581 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 13:38:22.387287 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.387275 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 13:38:22.387287 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.387285 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 13:38:22.390316 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.390303 2568 server.go:1268] "Started kubelet" Jul 24 13:38:22.390397 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.390361 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 13:38:22.391779 ip-10-0-138-248 systemd[1]: Started Kubernetes Kubelet. Jul 24 13:38:22.397280 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.397250 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-248.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 13:38:22.397378 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.397281 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 13:38:22.397508 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.397475 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 13:38:22.397654 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.397555 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 13:38:22.397727 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.397664 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 13:38:22.398028 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.398011 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 13:38:22.399327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.399308 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 13:38:22.399406 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.399314 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 24 13:38:22.404021 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.403108 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e4375ae6d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.390275693 +0000 UTC m=+0.330730405,LastTimestamp:2026-07-24 13:38:22.390275693 +0000 UTC m=+0.330730405,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.404390 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.404372 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 13:38:22.404483 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.404397 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 13:38:22.404535 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.404485 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:22.405196 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405023 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 13:38:22.405196 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405026 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 13:38:22.405196 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405068 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 13:38:22.405381 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405228 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 24 13:38:22.405381 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405239 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 24 13:38:22.405381 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405354 2568 factory.go:55] Registering systemd factory Jul 24 13:38:22.405381 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405369 2568 factory.go:223] Registration of the systemd container factory successfully Jul 24 13:38:22.405940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405923 2568 factory.go:153] Registering CRI-O factory Jul 24 13:38:22.405940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405942 2568 factory.go:223] Registration of the crio container factory successfully Jul 24 13:38:22.406077 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.405996 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 13:38:22.406077 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.406016 2568 factory.go:103] Registering Raw factory Jul 24 13:38:22.406077 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.406029 2568 manager.go:1196] Started watching for new ooms in manager Jul 24 13:38:22.406414 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.406400 2568 manager.go:349] Starting recovery of all containers Jul 24 13:38:22.408745 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.408650 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-248.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 13:38:22.408844 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.408792 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 13:38:22.412729 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.412682 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 13:38:22.416188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.416164 2568 manager.go:354] Recovery completed Jul 24 13:38:22.420207 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.420194 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.421907 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.421891 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.421964 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.421918 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.421964 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.421927 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.422385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.422363 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 13:38:22.422385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.422375 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 13:38:22.422475 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.422392 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 24 13:38:22.422475 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.422417 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.423699 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.423683 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.423792 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.423707 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.423792 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.423719 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.423792 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.423773 2568 policy_none.go:49] "None policy: Start" Jul 24 13:38:22.423792 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.423787 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 13:38:22.423978 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.423798 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 13:38:22.423978 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.423780 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e45584d63 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-248.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.421904739 +0000 UTC m=+0.362359450,LastTimestamp:2026-07-24 13:38:22.421904739 +0000 UTC m=+0.362359450,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.424618 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.424607 2568 policy_none.go:47] "Start" Jul 24 13:38:22.434296 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.434233 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e4558916d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-248.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.421922157 +0000 UTC m=+0.362376868,LastTimestamp:2026-07-24 13:38:22.421922157 +0000 UTC m=+0.362376868,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.444867 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.444801 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e4558b320 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-248.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.421930784 +0000 UTC m=+0.362385495,LastTimestamp:2026-07-24 13:38:22.421930784 +0000 UTC m=+0.362385495,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.457382 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.457321 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-248.ec2.internal.18c53d5e45584d63\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e45584d63 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-248.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.421904739 +0000 UTC m=+0.362359450,LastTimestamp:2026-07-24 13:38:22.423697482 +0000 UTC m=+0.364152198,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.465295 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465282 2568 manager.go:337] "Starting Device Plugin manager" Jul 24 13:38:22.465352 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.465309 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 13:38:22.465352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465318 2568 server.go:87] "Starting device plugin registration server" Jul 24 13:38:22.465512 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465502 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 13:38:22.465565 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465514 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 13:38:22.465799 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465622 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 13:38:22.465799 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465703 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 13:38:22.465799 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.465712 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 13:38:22.465947 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.465842 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-248.ec2.internal.18c53d5e4558916d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e4558916d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-248.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.421922157 +0000 UTC m=+0.362376868,LastTimestamp:2026-07-24 13:38:22.423712767 +0000 UTC m=+0.364167478,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.466104 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.466041 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 13:38:22.466104 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.466097 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:22.472514 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.472498 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nw6h" Jul 24 13:38:22.474115 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.474064 2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-248.ec2.internal.18c53d5e4558b320\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e4558b320 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-248.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.421930784 +0000 UTC m=+0.362385495,LastTimestamp:2026-07-24 13:38:22.423723123 +0000 UTC m=+0.364177834,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.480471 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.480455 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nw6h" Jul 24 13:38:22.481341 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.481288 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-248.ec2.internal.18c53d5e480f2459 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-248.ec2.internal,UID:ip-10-0-138-248.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-248.ec2.internal,},FirstTimestamp:2026-07-24 13:38:22.467441753 +0000 UTC m=+0.407896451,LastTimestamp:2026-07-24 13:38:22.467441753 +0000 UTC m=+0.407896451,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-248.ec2.internal,}" Jul 24 13:38:22.522887 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.522865 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 13:38:22.524879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.522900 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 13:38:22.524879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.522918 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 13:38:22.524879 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.522946 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 13:38:22.532175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.532161 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:22.565895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.565879 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.567259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.567246 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.567334 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.567272 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.567334 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.567282 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.567334 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.567299 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.574589 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.574574 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.574679 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.574596 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-248.ec2.internal\": node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:22.600479 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.600420 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:22.623557 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.623535 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal"] Jul 24 13:38:22.623610 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.623584 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.624278 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.624258 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.624347 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.624288 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.624347 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.624301 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.626544 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.626532 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.626685 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.626672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.626720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.626699 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.627329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.627314 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.627412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.627340 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.627412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.627342 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.627412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.627354 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.627412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.627363 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.627412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.627373 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.629480 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.629466 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.629521 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.629491 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:22.630345 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.630332 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:22.630415 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.630354 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:22.630415 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.630366 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:22.653319 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.653306 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-248.ec2.internal\" not found" node="ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.657613 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.657597 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-248.ec2.internal\" not found" node="ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.700853 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:22.700836 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:22.707180 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.707163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/771381fabc06c5e919c1029bae495d3d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-248.ec2.internal\" (UID: \"771381fabc06c5e919c1029bae495d3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.707229 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.707189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f539b6bf924b8ab216aac6d1f96cd571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal\" (UID: \"f539b6bf924b8ab216aac6d1f96cd571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:22.707229 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:22.707206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f539b6bf924b8ab216aac6d1f96cd571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal\" (UID: \"f539b6bf924b8ab216aac6d1f96cd571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.131299 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.131274 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.137606 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.137590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f539b6bf924b8ab216aac6d1f96cd571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal\" (UID: \"f539b6bf924b8ab216aac6d1f96cd571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.137650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.137613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f539b6bf924b8ab216aac6d1f96cd571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal\" (UID: \"f539b6bf924b8ab216aac6d1f96cd571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.137650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.137628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/771381fabc06c5e919c1029bae495d3d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-248.ec2.internal\" (UID: \"771381fabc06c5e919c1029bae495d3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.137738 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.137663 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/771381fabc06c5e919c1029bae495d3d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-248.ec2.internal\" (UID: \"771381fabc06c5e919c1029bae495d3d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.137738 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.137698 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f539b6bf924b8ab216aac6d1f96cd571-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal\" (UID: \"f539b6bf924b8ab216aac6d1f96cd571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.137806 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.137698 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f539b6bf924b8ab216aac6d1f96cd571-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal\" (UID: \"f539b6bf924b8ab216aac6d1f96cd571\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.232072 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.231993 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.285455 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.285430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.291348 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.291232 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.332419 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.332396 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.432901 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.432870 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.533467 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.533413 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.633830 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.633808 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.676250 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.676230 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 13:38:24.676601 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.676362 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 13:38:24.734067 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.734049 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.801197 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.801142 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 13:38:24.804007 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.803986 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientMemory" Jul 24 13:38:24.804101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.804021 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 13:38:24.804101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.804032 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeHasSufficientPID" Jul 24 13:38:24.817033 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.817000 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 13:33:23 +0000 UTC" deadline="2028-03-12 15:00:00.301485862 +0000 UTC" Jul 24 13:38:24.817033 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.817028 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14329h21m35.484462086s" Jul 24 13:38:24.834680 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:24.834656 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-248.ec2.internal\" not found" Jul 24 13:38:24.839292 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:24.839267 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771381fabc06c5e919c1029bae495d3d.slice/crio-4aab90f3af5a90bf870065981bb3e5a5d8181c16fb0ab9390bbfba17334f9b16 WatchSource:0}: Error finding container 4aab90f3af5a90bf870065981bb3e5a5d8181c16fb0ab9390bbfba17334f9b16: Status 404 returned error can't find the container with id 4aab90f3af5a90bf870065981bb3e5a5d8181c16fb0ab9390bbfba17334f9b16 Jul 24 13:38:24.843107 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.843095 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:38:24.855581 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.855542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" event={"ID":"771381fabc06c5e919c1029bae495d3d","Type":"ContainerStarted","Data":"4aab90f3af5a90bf870065981bb3e5a5d8181c16fb0ab9390bbfba17334f9b16"} Jul 24 13:38:24.882110 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.882088 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:24.935177 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.935157 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.949167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.949148 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 13:38:24.950593 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.950581 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" Jul 24 13:38:24.959229 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:24.959213 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 13:38:25.033830 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.033810 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:25.097091 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.097074 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:25.099044 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:25.099022 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf539b6bf924b8ab216aac6d1f96cd571.slice/crio-00cb5587e42a09ebe1d2821e5adf994d5b1cf49eb88dfd13c287333b61cd543d WatchSource:0}: Error finding container 00cb5587e42a09ebe1d2821e5adf994d5b1cf49eb88dfd13c287333b61cd543d: Status 404 returned error can't find the container with id 00cb5587e42a09ebe1d2821e5adf994d5b1cf49eb88dfd13c287333b61cd543d Jul 24 13:38:25.711621 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.711542 2568 apiserver.go:52] "Watching apiserver" Jul 24 13:38:25.721655 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.721625 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 13:38:25.723939 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.723910 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sw9nh","kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xrzkc","openshift-dns/node-resolver-c5q46","openshift-image-registry/node-ca-kcsmh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal","openshift-multus/multus-additional-cni-plugins-wtfp4","openshift-multus/multus-x4p7j","kube-system/global-pull-secret-syncer-vnqvq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf","openshift-multus/network-metrics-daemon-tjr2x","openshift-network-diagnostics/network-check-target-7j8ww","openshift-network-operator/iptables-alerter-x6rjq","openshift-ovn-kubernetes/ovnkube-node-2llgg"] Jul 24 13:38:25.726487 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.726465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.729389 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.729366 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.729561 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.729543 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.729634 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.729554 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.730016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.729994 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7b7ts\"" Jul 24 13:38:25.731632 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.731149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.731632 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.731572 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.732496 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.731924 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 13:38:25.732496 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.731950 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.732496 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.732240 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 13:38:25.732496 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.732294 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ppwbc\"" Jul 24 13:38:25.733224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.733204 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 13:38:25.733598 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.733579 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 13:38:25.733673 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.733658 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.733948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.733926 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2nk7k\"" Jul 24 13:38:25.735641 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.735619 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 13:38:25.737365 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.737056 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.737365 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.737316 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.737472 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.737421 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jpd7w\"" Jul 24 13:38:25.737937 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.737665 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.739311 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.739292 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.741746 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.741018 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 13:38:25.741746 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.741243 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.741746 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.741446 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9z29k\"" Jul 24 13:38:25.741746 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.741549 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.742440 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.742116 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.742440 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.742206 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:25.743498 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.743321 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 13:38:25.743498 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.743380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ngqml\"" Jul 24 13:38:25.745005 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.744978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-tuned\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c249f343-293f-40f2-afc1-8ca76ee1b617-tmp\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745661 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-k8s-cni-cncf-io\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-etc-kubernetes\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-system-cni-dir\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-modprobe-d\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.745863 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysconfig\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745910 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-conf-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-daemon-config\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.745972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6rpb\" (UniqueName: \"kubernetes.io/projected/241fa270-01bf-44d5-a857-22457556bd4f-kube-api-access-n6rpb\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746032 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-systemd\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-lib-modules\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-os-release\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-os-release\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-socket-dir-parent\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746316 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-cni-bin\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-kubelet\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746380 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysctl-d\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746408 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlcrb\" (UniqueName: \"kubernetes.io/projected/c249f343-293f-40f2-afc1-8ca76ee1b617-kube-api-access-vlcrb\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-system-cni-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be8f9fc-7757-4af3-8897-5c7866503c5f-cni-binary-copy\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-run\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.747961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-sys\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-var-lib-kubelet\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746644 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-multus-certs\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9gg\" (UniqueName: \"kubernetes.io/projected/6be8f9fc-7757-4af3-8897-5c7866503c5f-kube-api-access-2x9gg\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-cnibin\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746850 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvmv\" (UniqueName: \"kubernetes.io/projected/ef5c0207-b134-4604-8b0e-14ea1481ee07-kube-api-access-zfvmv\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysctl-conf\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.746946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-cni-multus\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747091 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e83441c2-b725-4664-91d0-5e915d2ba6c1-hosts-file\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrvk\" (UniqueName: \"kubernetes.io/projected/e83441c2-b725-4664-91d0-5e915d2ba6c1-kube-api-access-pjrvk\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-hostroot\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83441c2-b725-4664-91d0-5e915d2ba6c1-tmp-dir\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241fa270-01bf-44d5-a857-22457556bd4f-host\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.748692 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747275 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/241fa270-01bf-44d5-a857-22457556bd4f-serviceca\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-netns\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-kubernetes\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-cni-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-cnibin\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.747485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-host\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.748883 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.749317 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2gdlg\"" Jul 24 13:38:25.749524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.749470 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 13:38:25.749909 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.749631 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.750439 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.750418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:25.750543 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.750514 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:25.751541 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.751357 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 13:38:25.752427 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.752319 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.752712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.752694 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:25.752805 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.752768 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:25.755088 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.755066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.757482 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.757459 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-578fx\"" Jul 24 13:38:25.757572 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.757547 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.757712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.757696 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 13:38:25.758312 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.758194 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.758312 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.758215 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.761505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.761485 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 13:38:25.761772 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.761755 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 13:38:25.761968 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.761952 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 13:38:25.762146 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.762130 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 13:38:25.762331 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.762316 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 13:38:25.762520 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.762504 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqtwv\"" Jul 24 13:38:25.762761 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.762745 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 13:38:25.773952 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.773932 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c6hck" Jul 24 13:38:25.784484 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.784462 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c6hck" Jul 24 13:38:25.837331 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.837310 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 13:38:25.848275 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848250 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-netns\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.848376 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-kubernetes\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848376 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848319 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:25.848376 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848343 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-netns\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.848376 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848344 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b39d2add-7297-4c19-9836-c1c9e7e44711-iptables-alerter-script\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-host\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848403 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-kubernetes\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-tuned\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c249f343-293f-40f2-afc1-8ca76ee1b617-tmp\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-k8s-cni-cncf-io\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848492 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-host\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-k8s-cni-cncf-io\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b70c494-5fd3-4783-a114-907bf3f3805e-kubelet-config\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysconfig\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-conf-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848621 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysconfig\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-conf-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.848696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848682 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7472c-bbd5-44be-9952-53f299a387e9-konnectivity-ca\") pod \"konnectivity-agent-sw9nh\" (UID: \"1eb7472c-bbd5-44be-9952-53f299a387e9\") " pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b70c494-5fd3-4783-a114-907bf3f3805e-dbus\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848718 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-systemd\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-os-release\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-os-release\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-socket-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848844 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-systemd\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-etc-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848868 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-os-release\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-multus-certs\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-os-release\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9gg\" (UniqueName: \"kubernetes.io/projected/6be8f9fc-7757-4af3-8897-5c7866503c5f-kube-api-access-2x9gg\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848961 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-cnibin\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-run-multus-certs\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.848988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvmv\" (UniqueName: \"kubernetes.io/projected/ef5c0207-b134-4604-8b0e-14ea1481ee07-kube-api-access-zfvmv\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-cnibin\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.849384 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-cni-bin\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-run-netns\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-var-lib-kubelet\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849178 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-cni-bin\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1500c1b-a17a-4f28-8524-f921189cdebb-ovn-node-metrics-cert\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849259 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-var-lib-kubelet\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-registration-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849340 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-slash\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849414 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39d2add-7297-4c19-9836-c1c9e7e44711-host-slash\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.850202 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-systemd-units\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-var-lib-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-log-socket\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-cni-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-cnibin\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849573 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftm98\" (UniqueName: \"kubernetes.io/projected/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-kube-api-access-ftm98\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849599 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvw4\" (UniqueName: \"kubernetes.io/projected/b39d2add-7297-4c19-9836-c1c9e7e44711-kube-api-access-xfvw4\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-cni-bin\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-env-overrides\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-etc-kubernetes\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849694 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-system-cni-dir\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849708 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-cni-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849716 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-cnibin\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849718 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-systemd\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-cni-netd\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849777 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-etc-kubernetes\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.850966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849789 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef5c0207-b134-4604-8b0e-14ea1481ee07-system-cni-dir\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-modprobe-d\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-daemon-config\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6rpb\" (UniqueName: \"kubernetes.io/projected/241fa270-01bf-44d5-a857-22457556bd4f-kube-api-access-n6rpb\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849866 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-modprobe-d\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-sys-fs\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-lib-modules\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849957 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-device-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.849981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83441c2-b725-4664-91d0-5e915d2ba6c1-tmp-dir\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241fa270-01bf-44d5-a857-22457556bd4f-host\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/241fa270-01bf-44d5-a857-22457556bd4f-serviceca\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-lib-modules\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850087 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/241fa270-01bf-44d5-a857-22457556bd4f-host\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850107 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-socket-dir-parent\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-socket-dir-parent\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-kubelet\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysctl-d\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlcrb\" (UniqueName: \"kubernetes.io/projected/c249f343-293f-40f2-afc1-8ca76ee1b617-kube-api-access-vlcrb\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.851720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-system-cni-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-run\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-sys\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e83441c2-b725-4664-91d0-5e915d2ba6c1-hosts-file\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrvk\" (UniqueName: \"kubernetes.io/projected/e83441c2-b725-4664-91d0-5e915d2ba6c1-kube-api-access-pjrvk\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850330 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e83441c2-b725-4664-91d0-5e915d2ba6c1-tmp-dir\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67ps\" (UniqueName: \"kubernetes.io/projected/1ccb971b-88bb-4e91-aa7a-4006dca11af0-kube-api-access-z67ps\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be8f9fc-7757-4af3-8897-5c7866503c5f-cni-binary-copy\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-system-cni-dir\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-kubelet\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850450 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysctl-conf\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850458 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysctl-d\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-cni-multus\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/241fa270-01bf-44d5-a857-22457556bd4f-serviceca\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850528 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-run\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850347 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be8f9fc-7757-4af3-8897-5c7866503c5f-multus-daemon-config\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.852604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-sys\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-node-log\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850496 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-kubelet\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-run-ovn-kubernetes\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850627 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-ovnkube-config\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-host-var-lib-cni-multus\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-ovnkube-script-lib\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850654 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-sysctl-conf\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850691 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-hostroot\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850719 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1eb7472c-bbd5-44be-9952-53f299a387e9-agent-certs\") pod \"konnectivity-agent-sw9nh\" (UID: \"1eb7472c-bbd5-44be-9952-53f299a387e9\") " pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850782 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e83441c2-b725-4664-91d0-5e915d2ba6c1-hosts-file\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-ovn\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546tn\" (UniqueName: \"kubernetes.io/projected/e1500c1b-a17a-4f28-8524-f921189cdebb-kube-api-access-546tn\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.850876 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be8f9fc-7757-4af3-8897-5c7866503c5f-hostroot\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.851109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef5c0207-b134-4604-8b0e-14ea1481ee07-cni-binary-copy\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.853336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.851127 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be8f9fc-7757-4af3-8897-5c7866503c5f-cni-binary-copy\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.854122 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.852542 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c249f343-293f-40f2-afc1-8ca76ee1b617-etc-tuned\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.854122 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.852796 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c249f343-293f-40f2-afc1-8ca76ee1b617-tmp\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.861453 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.861384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrvk\" (UniqueName: \"kubernetes.io/projected/e83441c2-b725-4664-91d0-5e915d2ba6c1-kube-api-access-pjrvk\") pod \"node-resolver-c5q46\" (UID: \"e83441c2-b725-4664-91d0-5e915d2ba6c1\") " pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:25.861745 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.861673 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6rpb\" (UniqueName: \"kubernetes.io/projected/241fa270-01bf-44d5-a857-22457556bd4f-kube-api-access-n6rpb\") pod \"node-ca-kcsmh\" (UID: \"241fa270-01bf-44d5-a857-22457556bd4f\") " pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:25.861965 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.861878 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9gg\" (UniqueName: \"kubernetes.io/projected/6be8f9fc-7757-4af3-8897-5c7866503c5f-kube-api-access-2x9gg\") pod \"multus-x4p7j\" (UID: \"6be8f9fc-7757-4af3-8897-5c7866503c5f\") " pod="openshift-multus/multus-x4p7j" Jul 24 13:38:25.863570 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.862986 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" event={"ID":"f539b6bf924b8ab216aac6d1f96cd571","Type":"ContainerStarted","Data":"00cb5587e42a09ebe1d2821e5adf994d5b1cf49eb88dfd13c287333b61cd543d"} Jul 24 13:38:25.863570 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.863519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlcrb\" (UniqueName: \"kubernetes.io/projected/c249f343-293f-40f2-afc1-8ca76ee1b617-kube-api-access-vlcrb\") pod \"tuned-xrzkc\" (UID: \"c249f343-293f-40f2-afc1-8ca76ee1b617\") " pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:25.867100 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.867068 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvmv\" (UniqueName: \"kubernetes.io/projected/ef5c0207-b134-4604-8b0e-14ea1481ee07-kube-api-access-zfvmv\") pod \"multus-additional-cni-plugins-wtfp4\" (UID: \"ef5c0207-b134-4604-8b0e-14ea1481ee07\") " pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:25.951804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.951777 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-cni-bin\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.951960 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.951811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-env-overrides\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.951960 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.951835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-systemd\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.951960 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.951897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-cni-bin\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.951960 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.951945 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-cni-netd\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.951981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-sys-fs\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-cni-netd\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952036 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-systemd\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-device-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952044 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-sys-fs\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952094 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67ps\" (UniqueName: \"kubernetes.io/projected/1ccb971b-88bb-4e91-aa7a-4006dca11af0-kube-api-access-z67ps\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952105 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-device-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-kubelet\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952175 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952215 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-node-log\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-run-ovn-kubernetes\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952243 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952262 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-env-overrides\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-ovnkube-config\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-etc-selinux\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-node-log\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-ovnkube-script-lib\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952403 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-kubelet\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1eb7472c-bbd5-44be-9952-53f299a387e9-agent-certs\") pod \"konnectivity-agent-sw9nh\" (UID: \"1eb7472c-bbd5-44be-9952-53f299a387e9\") " pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-run-ovn-kubernetes\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-ovn\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.952592 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-run-ovn\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.952606 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546tn\" (UniqueName: \"kubernetes.io/projected/e1500c1b-a17a-4f28-8524-f921189cdebb-kube-api-access-546tn\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.952675 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:38:26.452655192 +0000 UTC m=+3.063134921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-ovnkube-config\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b39d2add-7297-4c19-9836-c1c9e7e44711-iptables-alerter-script\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b70c494-5fd3-4783-a114-907bf3f3805e-kubelet-config\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7472c-bbd5-44be-9952-53f299a387e9-konnectivity-ca\") pod \"konnectivity-agent-sw9nh\" (UID: \"1eb7472c-bbd5-44be-9952-53f299a387e9\") " pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.952930 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952933 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b70c494-5fd3-4783-a114-907bf3f3805e-dbus\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-socket-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.952971 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:26.4529573 +0000 UTC m=+3.063437030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.952999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-etc-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-run-netns\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.953329 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953069 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-socket-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953100 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-etc-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1500c1b-a17a-4f28-8524-f921189cdebb-ovn-node-metrics-cert\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8b70c494-5fd3-4783-a114-907bf3f3805e-kubelet-config\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1500c1b-a17a-4f28-8524-f921189cdebb-ovnkube-script-lib\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953152 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-run-netns\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-registration-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-slash\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953255 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39d2add-7297-4c19-9836-c1c9e7e44711-host-slash\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-systemd-units\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-var-lib-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953331 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-log-socket\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftm98\" (UniqueName: \"kubernetes.io/projected/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-kube-api-access-ftm98\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953347 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8b70c494-5fd3-4783-a114-907bf3f3805e-dbus\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:25.954151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvw4\" (UniqueName: \"kubernetes.io/projected/b39d2add-7297-4c19-9836-c1c9e7e44711-kube-api-access-xfvw4\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953401 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b39d2add-7297-4c19-9836-c1c9e7e44711-host-slash\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953422 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b39d2add-7297-4c19-9836-c1c9e7e44711-iptables-alerter-script\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953606 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-systemd-units\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953648 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-log-socket\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953706 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-var-lib-openvswitch\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1500c1b-a17a-4f28-8524-f921189cdebb-host-slash\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953793 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.953797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-registration-dir\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:25.955032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.954261 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1eb7472c-bbd5-44be-9952-53f299a387e9-konnectivity-ca\") pod \"konnectivity-agent-sw9nh\" (UID: \"1eb7472c-bbd5-44be-9952-53f299a387e9\") " pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.955482 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.955235 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1eb7472c-bbd5-44be-9952-53f299a387e9-agent-certs\") pod \"konnectivity-agent-sw9nh\" (UID: \"1eb7472c-bbd5-44be-9952-53f299a387e9\") " pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:25.955990 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.955966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1500c1b-a17a-4f28-8524-f921189cdebb-ovn-node-metrics-cert\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.959765 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.959702 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:25.959765 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.959744 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:25.959765 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.959757 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:25.959945 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:25.959815 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:26.459798729 +0000 UTC m=+3.070278470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:25.962155 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.962089 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-546tn\" (UniqueName: \"kubernetes.io/projected/e1500c1b-a17a-4f28-8524-f921189cdebb-kube-api-access-546tn\") pod \"ovnkube-node-2llgg\" (UID: \"e1500c1b-a17a-4f28-8524-f921189cdebb\") " pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:25.962580 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.962558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67ps\" (UniqueName: \"kubernetes.io/projected/1ccb971b-88bb-4e91-aa7a-4006dca11af0-kube-api-access-z67ps\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:25.963813 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.963797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvw4\" (UniqueName: \"kubernetes.io/projected/b39d2add-7297-4c19-9836-c1c9e7e44711-kube-api-access-xfvw4\") pod \"iptables-alerter-x6rjq\" (UID: \"b39d2add-7297-4c19-9836-c1c9e7e44711\") " pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:25.963938 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:25.963919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftm98\" (UniqueName: \"kubernetes.io/projected/e3ff5675-5e78-4ef3-a3ca-3cda583a07d1-kube-api-access-ftm98\") pod \"aws-ebs-csi-driver-node-qn4pf\" (UID: \"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:26.042900 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.042871 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" Jul 24 13:38:26.054237 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.054213 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x4p7j" Jul 24 13:38:26.064819 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.064804 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" Jul 24 13:38:26.074396 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.074381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c5q46" Jul 24 13:38:26.084014 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.083993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kcsmh" Jul 24 13:38:26.090567 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.090550 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:26.100117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.100095 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" Jul 24 13:38:26.109623 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.109607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x6rjq" Jul 24 13:38:26.115267 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.115246 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:26.243976 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.243906 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 13:38:26.458018 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.457979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:26.458188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.458037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:26.458188 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.458139 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:26.458292 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.458209 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:26.458292 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.458211 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:38:27.458192515 +0000 UTC m=+4.068672244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:26.458292 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.458270 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:27.458253181 +0000 UTC m=+4.068732910 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:26.558974 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.558901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:26.559120 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.559055 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:26.559120 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.559076 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:26.559120 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.559089 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:26.559233 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:26.559144 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:27.559127284 +0000 UTC m=+4.169607010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:26.787819 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.787781 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:33:25 +0000 UTC" deadline="2028-03-12 23:11:34.245155661 +0000 UTC" Jul 24 13:38:26.787819 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:26.787811 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14337h33m7.457348119s" Jul 24 13:38:26.958701 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.958672 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83441c2_b725_4664_91d0_5e915d2ba6c1.slice/crio-1736a623843db0a851040bb3e14101dafa7de10e4a4804153e196b6fa6fff8f2 WatchSource:0}: Error finding container 1736a623843db0a851040bb3e14101dafa7de10e4a4804153e196b6fa6fff8f2: Status 404 returned error can't find the container with id 1736a623843db0a851040bb3e14101dafa7de10e4a4804153e196b6fa6fff8f2 Jul 24 13:38:26.959249 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.959227 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241fa270_01bf_44d5_a857_22457556bd4f.slice/crio-8d5b784bf021eb0346614f21df58f4439ed62abf802b432c8a0f9a57e0ed9875 WatchSource:0}: Error finding container 8d5b784bf021eb0346614f21df58f4439ed62abf802b432c8a0f9a57e0ed9875: Status 404 returned error can't find the container with id 8d5b784bf021eb0346614f21df58f4439ed62abf802b432c8a0f9a57e0ed9875 Jul 24 13:38:26.960973 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.960935 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1500c1b_a17a_4f28_8524_f921189cdebb.slice/crio-a438c4109179e7fd9812a85a1388580e7b0004939b0f123b5d97a483eca926c7 WatchSource:0}: Error finding container a438c4109179e7fd9812a85a1388580e7b0004939b0f123b5d97a483eca926c7: Status 404 returned error can't find the container with id a438c4109179e7fd9812a85a1388580e7b0004939b0f123b5d97a483eca926c7 Jul 24 13:38:26.964449 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.964384 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be8f9fc_7757_4af3_8897_5c7866503c5f.slice/crio-54429eb030e436c911f1bdad675aaf21c22839dc1990f8b2d91aaf74a9b76238 WatchSource:0}: Error finding container 54429eb030e436c911f1bdad675aaf21c22839dc1990f8b2d91aaf74a9b76238: Status 404 returned error can't find the container with id 54429eb030e436c911f1bdad675aaf21c22839dc1990f8b2d91aaf74a9b76238 Jul 24 13:38:26.965747 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.965713 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5c0207_b134_4604_8b0e_14ea1481ee07.slice/crio-dbcb949d943d11e52e4dcb4212a11d3054df5821d7213ba8a4ed54bea9c0806e WatchSource:0}: Error finding container dbcb949d943d11e52e4dcb4212a11d3054df5821d7213ba8a4ed54bea9c0806e: Status 404 returned error can't find the container with id dbcb949d943d11e52e4dcb4212a11d3054df5821d7213ba8a4ed54bea9c0806e Jul 24 13:38:26.966540 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.966506 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb7472c_bbd5_44be_9952_53f299a387e9.slice/crio-8f594e5420d9c42d8b81611750f118d27f143c50192a6e6a05e2401ec09949d2 WatchSource:0}: Error finding container 8f594e5420d9c42d8b81611750f118d27f143c50192a6e6a05e2401ec09949d2: Status 404 returned error can't find the container with id 8f594e5420d9c42d8b81611750f118d27f143c50192a6e6a05e2401ec09949d2 Jul 24 13:38:26.968894 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.967496 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39d2add_7297_4c19_9836_c1c9e7e44711.slice/crio-7082ae8e06e86a12f2115e9e16b693683f3c2853c892d9d264764bd62418e0ed WatchSource:0}: Error finding container 7082ae8e06e86a12f2115e9e16b693683f3c2853c892d9d264764bd62418e0ed: Status 404 returned error can't find the container with id 7082ae8e06e86a12f2115e9e16b693683f3c2853c892d9d264764bd62418e0ed Jul 24 13:38:26.969781 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.969455 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc249f343_293f_40f2_afc1_8ca76ee1b617.slice/crio-bb6adb78719ff32cf35ff3da6c247b56e8d3dfca153ecf6ac84219c886cdf11b WatchSource:0}: Error finding container bb6adb78719ff32cf35ff3da6c247b56e8d3dfca153ecf6ac84219c886cdf11b: Status 404 returned error can't find the container with id bb6adb78719ff32cf35ff3da6c247b56e8d3dfca153ecf6ac84219c886cdf11b Jul 24 13:38:26.970069 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:26.970044 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ff5675_5e78_4ef3_a3ca_3cda583a07d1.slice/crio-92b3c5fbb46830c768cea1be0ac3b0fcb2a1753a63b3af738b5e1c640dee251f WatchSource:0}: Error finding container 92b3c5fbb46830c768cea1be0ac3b0fcb2a1753a63b3af738b5e1c640dee251f: Status 404 returned error can't find the container with id 92b3c5fbb46830c768cea1be0ac3b0fcb2a1753a63b3af738b5e1c640dee251f Jul 24 13:38:27.465604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.465353 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:27.465768 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.465622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:27.465815 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.465791 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:27.465815 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.465803 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:27.465884 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.465855 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:38:29.465835792 +0000 UTC m=+6.076315535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:27.465884 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.465872 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:29.465860916 +0000 UTC m=+6.076340655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:27.566611 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.566575 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:27.566822 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.566805 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:27.566896 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.566833 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:27.566896 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.566846 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:27.566992 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.566902 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:29.566884829 +0000 UTC m=+6.177364558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:27.788360 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.788254 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 13:33:25 +0000 UTC" deadline="2028-01-15 21:57:12.525911417 +0000 UTC" Jul 24 13:38:27.788360 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.788311 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12968h18m44.737604595s" Jul 24 13:38:27.855216 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.855185 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:27.855439 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.855322 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:27.855813 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.855793 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:27.855906 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.855891 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:27.855978 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.855937 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:27.856029 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:27.856016 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:27.872470 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.872428 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" event={"ID":"771381fabc06c5e919c1029bae495d3d","Type":"ContainerStarted","Data":"d8dcefba1bd4ac110080b7012871f323fbc6712874b52ba5d799170e7f610aa1"} Jul 24 13:38:27.879569 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.879533 2568 generic.go:356] "Generic (PLEG): container finished" podID="f539b6bf924b8ab216aac6d1f96cd571" containerID="bb579f316adfe37f385215fad14dfbe81fada0482692dfd893f82414b4f89025" exitCode=0 Jul 24 13:38:27.880176 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.880149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" event={"ID":"f539b6bf924b8ab216aac6d1f96cd571","Type":"ContainerDied","Data":"bb579f316adfe37f385215fad14dfbe81fada0482692dfd893f82414b4f89025"} Jul 24 13:38:27.886852 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.886818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" event={"ID":"c249f343-293f-40f2-afc1-8ca76ee1b617","Type":"ContainerStarted","Data":"bb6adb78719ff32cf35ff3da6c247b56e8d3dfca153ecf6ac84219c886cdf11b"} Jul 24 13:38:27.888617 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.888570 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-248.ec2.internal" podStartSLOduration=3.888550708 podStartE2EDuration="3.888550708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:27.887753494 +0000 UTC m=+4.498233243" watchObservedRunningTime="2026-07-24 13:38:27.888550708 +0000 UTC m=+4.499030457" Jul 24 13:38:27.895324 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.895289 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerStarted","Data":"dbcb949d943d11e52e4dcb4212a11d3054df5821d7213ba8a4ed54bea9c0806e"} Jul 24 13:38:27.913816 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.913700 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4p7j" event={"ID":"6be8f9fc-7757-4af3-8897-5c7866503c5f","Type":"ContainerStarted","Data":"54429eb030e436c911f1bdad675aaf21c22839dc1990f8b2d91aaf74a9b76238"} Jul 24 13:38:27.918657 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.918621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" event={"ID":"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1","Type":"ContainerStarted","Data":"92b3c5fbb46830c768cea1be0ac3b0fcb2a1753a63b3af738b5e1c640dee251f"} Jul 24 13:38:27.920575 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.920547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sw9nh" event={"ID":"1eb7472c-bbd5-44be-9952-53f299a387e9","Type":"ContainerStarted","Data":"8f594e5420d9c42d8b81611750f118d27f143c50192a6e6a05e2401ec09949d2"} Jul 24 13:38:27.922624 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.922595 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6rjq" event={"ID":"b39d2add-7297-4c19-9836-c1c9e7e44711","Type":"ContainerStarted","Data":"7082ae8e06e86a12f2115e9e16b693683f3c2853c892d9d264764bd62418e0ed"} Jul 24 13:38:27.928994 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.928960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"a438c4109179e7fd9812a85a1388580e7b0004939b0f123b5d97a483eca926c7"} Jul 24 13:38:27.933688 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.933636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcsmh" event={"ID":"241fa270-01bf-44d5-a857-22457556bd4f","Type":"ContainerStarted","Data":"8d5b784bf021eb0346614f21df58f4439ed62abf802b432c8a0f9a57e0ed9875"} Jul 24 13:38:27.937513 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:27.937480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c5q46" event={"ID":"e83441c2-b725-4664-91d0-5e915d2ba6c1","Type":"ContainerStarted","Data":"1736a623843db0a851040bb3e14101dafa7de10e4a4804153e196b6fa6fff8f2"} Jul 24 13:38:28.965390 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:28.964804 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" event={"ID":"f539b6bf924b8ab216aac6d1f96cd571","Type":"ContainerStarted","Data":"d42a86b1cff2deb8876ec2558ab5270b81e20173482e90daaac01695fba9ad32"} Jul 24 13:38:29.482954 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:29.482915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:29.483156 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:29.482967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:29.483156 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.483089 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:29.483156 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.483151 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:33.483133504 +0000 UTC m=+10.093613230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:29.483760 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.483529 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:29.483760 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.483579 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:38:33.483565276 +0000 UTC m=+10.094045005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:29.584080 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:29.584041 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:29.584251 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.584223 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:29.584251 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.584243 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:29.584372 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.584254 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:29.584372 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.584317 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:33.584299469 +0000 UTC m=+10.194779196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:29.854638 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:29.854558 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:29.854837 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:29.854570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:29.854837 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.854745 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:29.854965 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:29.854891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:29.855017 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.854992 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:29.855133 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:29.855077 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:31.853671 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:31.853188 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:31.853671 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:31.853315 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:31.853671 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:31.853351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:31.853671 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:31.853475 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:31.853671 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:31.853539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:31.853671 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:31.853627 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:33.516308 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:33.516262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:33.516808 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:33.516323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:33.516808 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.516464 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:33.516808 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.516497 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:33.516808 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.516554 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:38:41.516530668 +0000 UTC m=+18.127010403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:33.516808 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.516578 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:41.516567234 +0000 UTC m=+18.127046968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:33.617488 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:33.617447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:33.617695 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.617621 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:33.617695 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.617642 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:33.617695 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.617652 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:33.617867 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.617712 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:41.61769387 +0000 UTC m=+18.228173614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:33.855206 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:33.854692 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:33.855206 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.854840 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:33.855435 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:33.855238 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:33.855435 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.855344 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:33.855435 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:33.855395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:33.855556 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:33.855460 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:35.854182 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:35.854140 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:35.854651 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:35.854152 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:35.854651 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:35.854279 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:35.854651 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:35.854388 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:35.854651 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:35.854556 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:35.854651 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:35.854636 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:37.853323 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:37.853289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:37.853323 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:37.853319 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:37.853870 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:37.853287 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:37.853870 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:37.853430 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:37.853870 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:37.853514 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:37.853870 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:37.853589 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:39.854243 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:39.854203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:39.854773 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:39.854203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:39.854773 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:39.854347 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:39.854773 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:39.854452 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:39.854773 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:39.854203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:39.854773 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:39.854558 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:41.576953 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:41.576915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:41.576953 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:41.576955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:41.577389 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.577075 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:41.577389 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.577123 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:41.577389 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.577144 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:38:57.577127191 +0000 UTC m=+34.187606918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:41.577389 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.577173 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:57.577160439 +0000 UTC m=+34.187640166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:41.678266 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:41.678229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:41.678477 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.678358 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:41.678477 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.678372 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:41.678477 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.678381 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:41.678477 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.678429 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:57.67841626 +0000 UTC m=+34.288895985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:41.853969 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:41.853877 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:41.854132 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:41.854012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:41.854132 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.854072 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:41.854132 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.854095 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:41.854292 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:41.854132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:41.854292 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:41.854197 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:43.854574 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:43.854525 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:43.855101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:43.854632 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:43.855101 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:43.854645 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:43.855101 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:43.854840 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:43.855101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:43.854861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:43.855101 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:43.854983 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:45.853377 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:45.853218 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:45.853377 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:45.853346 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:45.854090 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:45.853424 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:45.854090 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:45.853510 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:45.854090 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:45.853637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:45.854914 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:45.853713 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:46.001916 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.001677 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" event={"ID":"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1","Type":"ContainerStarted","Data":"ef3dfea703d990d02dd35f95617ce7a807801e6520699b5e3e39dd9449dd2d4a"} Jul 24 13:38:46.003219 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.003184 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sw9nh" event={"ID":"1eb7472c-bbd5-44be-9952-53f299a387e9","Type":"ContainerStarted","Data":"7ec444a545c1f934bdafb2420d3d045d49376f9bc24b92117fd435164a4331d2"} Jul 24 13:38:46.005485 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.005458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"902c870e1ac6362a74db8a24c676954a1776709e96152a0d869f898fa38e9a6a"} Jul 24 13:38:46.005615 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.005542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"3e9f1ffa056b7c123e3153924d7794d6f18be0b30241382a8d4b719ccc818282"} Jul 24 13:38:46.005615 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.005592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"9eeb5972d110b50231da28847d08f78007d6dc082a61b217e19fe72594add474"} Jul 24 13:38:46.006841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.006818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kcsmh" event={"ID":"241fa270-01bf-44d5-a857-22457556bd4f","Type":"ContainerStarted","Data":"74b962670e833d64041f885d5a5711f76cb2fd671a57c6328f9b1e34ca3b4397"} Jul 24 13:38:46.008069 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.008048 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c5q46" event={"ID":"e83441c2-b725-4664-91d0-5e915d2ba6c1","Type":"ContainerStarted","Data":"2e50d97fe506a4555a833519c23f95c857a69565a8273c34cbf89be9ee329d92"} Jul 24 13:38:46.009332 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.009311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" event={"ID":"c249f343-293f-40f2-afc1-8ca76ee1b617","Type":"ContainerStarted","Data":"23a564674488dcbb95f3fa62a0fcb8b3fb0b9b6b1b9d9928197b568ddd1d7ab1"} Jul 24 13:38:46.010889 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.010869 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef5c0207-b134-4604-8b0e-14ea1481ee07" containerID="adfdb682a63d27c62dd26f77d017897d5b7ef74395bab3b2826097db5d4b2020" exitCode=0 Jul 24 13:38:46.011097 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.010940 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerDied","Data":"adfdb682a63d27c62dd26f77d017897d5b7ef74395bab3b2826097db5d4b2020"} Jul 24 13:38:46.012528 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.012507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x4p7j" event={"ID":"6be8f9fc-7757-4af3-8897-5c7866503c5f","Type":"ContainerStarted","Data":"618a7fe0c038dae9f3de389df4b4b2b2cb9d5ef54715c3978b3dae64126c77a3"} Jul 24 13:38:46.031305 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.031258 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-248.ec2.internal" podStartSLOduration=22.031239368 podStartE2EDuration="22.031239368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:28.98114806 +0000 UTC m=+5.591627808" watchObservedRunningTime="2026-07-24 13:38:46.031239368 +0000 UTC m=+22.641719115" Jul 24 13:38:46.031784 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.031757 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sw9nh" podStartSLOduration=11.260824576 podStartE2EDuration="22.031751024s" totalImagesPullingTime="10.770926448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:46.031088041 +0000 UTC m=+22.641567787" watchObservedRunningTime="2026-07-24 13:38:46.031751024 +0000 UTC m=+22.642230762" Jul 24 13:38:46.080912 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.080863 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xrzkc" podStartSLOduration=4.452582621 podStartE2EDuration="23.080844761s" totalImagesPullingTime="18.62826214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:46.053394135 +0000 UTC m=+22.663873881" watchObservedRunningTime="2026-07-24 13:38:46.080844761 +0000 UTC m=+22.691324504" Jul 24 13:38:46.081206 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.081175 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-x4p7j" podStartSLOduration=4.393821209 podStartE2EDuration="23.081165761s" totalImagesPullingTime="18.687344552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:46.080977298 +0000 UTC m=+22.691457027" watchObservedRunningTime="2026-07-24 13:38:46.081165761 +0000 UTC m=+22.691645506" Jul 24 13:38:46.183969 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.183918 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kcsmh" podStartSLOduration=3.5472744560000002 podStartE2EDuration="22.183902563s" totalImagesPullingTime="18.636628107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:46.1830824 +0000 UTC m=+22.793562146" watchObservedRunningTime="2026-07-24 13:38:46.183902563 +0000 UTC m=+22.794382303" Jul 24 13:38:46.214236 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:46.214173 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c5q46" podStartSLOduration=3.576518901 podStartE2EDuration="22.214155866s" totalImagesPullingTime="18.637636965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:46.214014488 +0000 UTC m=+22.824494235" watchObservedRunningTime="2026-07-24 13:38:46.214155866 +0000 UTC m=+22.824635611" Jul 24 13:38:47.016252 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.016222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x6rjq" event={"ID":"b39d2add-7297-4c19-9836-c1c9e7e44711","Type":"ContainerStarted","Data":"ea2dac04584fe91a4fa8a1d19c51efb880adc4f5acd30e12b86fb1257525c8cd"} Jul 24 13:38:47.019212 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.019177 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"828cf42f1d38da104968b28aa3e96917f48e2e13c019261d0af996cc957b1480"} Jul 24 13:38:47.019366 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.019253 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"7225c4945210f6ea8a3a9b235e86ba77dbec1ae30cf29279a57386def46e3ce6"} Jul 24 13:38:47.019366 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.019297 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"fdb031035ce5ede3cc8d6f1839e54fbb9e2de7e3ddbbff58a13503beab2c2fe7"} Jul 24 13:38:47.366368 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.366341 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 13:38:47.807881 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.807774 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T13:38:47.366359345Z","UUID":"b15ad3f0-8cee-4981-a4a2-521c3e66c85c","Handler":null,"Name":"","Endpoint":""} Jul 24 13:38:47.809448 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.809421 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 13:38:47.809589 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.809466 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 13:38:47.853873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.853846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:47.853873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.853865 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:47.854127 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:47.853968 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:47.854127 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:47.854057 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:47.854127 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:47.854120 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:47.854264 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:47.854197 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:48.024141 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:48.024100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" event={"ID":"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1","Type":"ContainerStarted","Data":"60a2b54e200ad460cbddddffe80b8d368126716d392110666480c373618083f7"} Jul 24 13:38:48.393036 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:48.392981 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:48.394350 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:48.394324 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:48.415804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:48.415755 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x6rjq" podStartSLOduration=5.794200371 podStartE2EDuration="24.415720806s" totalImagesPullingTime="18.621520435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:47.034766183 +0000 UTC m=+23.645245930" watchObservedRunningTime="2026-07-24 13:38:48.415720806 +0000 UTC m=+25.026200552" Jul 24 13:38:49.028792 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:49.028757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"297f9219a0c6e06b795e9b2167d80f841aaeb4b6a47d311b27a913fe1193d0ad"} Jul 24 13:38:49.029199 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:49.029027 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:49.029559 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:49.029545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sw9nh" Jul 24 13:38:49.853959 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:49.853714 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:49.854160 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:49.853715 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:49.854160 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:49.853777 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:49.854160 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:49.854080 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:49.854326 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:49.854161 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:49.854326 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:49.854234 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:50.033537 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:50.033500 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" event={"ID":"e3ff5675-5e78-4ef3-a3ca-3cda583a07d1","Type":"ContainerStarted","Data":"213a9f0a73544219bea1d035bd3cd59081974e878ca1af6134175208fd5839ab"} Jul 24 13:38:50.056059 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:50.055997 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qn4pf" podStartSLOduration=4.499283998 podStartE2EDuration="26.05597319s" totalImagesPullingTime="21.556689192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:50.054931011 +0000 UTC m=+26.665410757" watchObservedRunningTime="2026-07-24 13:38:50.05597319 +0000 UTC m=+26.666452940" Jul 24 13:38:51.040146 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.040107 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" event={"ID":"e1500c1b-a17a-4f28-8524-f921189cdebb","Type":"ContainerStarted","Data":"e7f2332f0b884cb951eb4021989c652f0ff3e47feaab9f880038807d5e562f38"} Jul 24 13:38:51.041107 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.040430 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:51.041107 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.040487 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:51.042218 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.042187 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerStarted","Data":"b1bef98a6672c741236e4346b81987b24eb15938f03dc34e6773726c587b1d6a"} Jul 24 13:38:51.057353 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.057120 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:51.067298 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.067250 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" podStartSLOduration=8.371299228 podStartE2EDuration="27.067234762s" totalImagesPullingTime="18.695935534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:38:51.066842471 +0000 UTC m=+27.677322217" watchObservedRunningTime="2026-07-24 13:38:51.067234762 +0000 UTC m=+27.677714508" Jul 24 13:38:51.853762 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.853708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:51.853981 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.853772 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:51.853981 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:51.853708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:51.853981 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:51.853852 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:51.853981 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:51.853922 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:51.854125 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:51.854013 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:52.045682 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.045645 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef5c0207-b134-4604-8b0e-14ea1481ee07" containerID="b1bef98a6672c741236e4346b81987b24eb15938f03dc34e6773726c587b1d6a" exitCode=0 Jul 24 13:38:52.046151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.045740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerDied","Data":"b1bef98a6672c741236e4346b81987b24eb15938f03dc34e6773726c587b1d6a"} Jul 24 13:38:52.046292 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.046242 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:52.063314 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.063285 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:38:52.930768 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.930734 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnqvq"] Jul 24 13:38:52.930935 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.930861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:52.930997 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:52.930971 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:52.931506 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.931483 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7j8ww"] Jul 24 13:38:52.931590 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.931560 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:52.931644 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:52.931626 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:52.934453 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.934420 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tjr2x"] Jul 24 13:38:52.934569 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:52.934525 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:52.934632 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:52.934613 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:54.052667 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:54.052630 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef5c0207-b134-4604-8b0e-14ea1481ee07" containerID="d3fdaeecb3cf0db4c9c0ac764741bebfcc73245ad6acac5571323283c31a49b9" exitCode=0 Jul 24 13:38:54.053138 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:54.052707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerDied","Data":"d3fdaeecb3cf0db4c9c0ac764741bebfcc73245ad6acac5571323283c31a49b9"} Jul 24 13:38:54.853865 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:54.853618 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:54.854023 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:54.853655 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:54.854023 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:54.853902 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:54.854023 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:54.853990 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:54.854023 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:54.853655 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:54.854205 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:54.854098 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:55.059117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:55.059087 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef5c0207-b134-4604-8b0e-14ea1481ee07" containerID="888c19258cdb54cfe53bfdb792f442323dbe39eac7656ed0d197904f6567cd68" exitCode=0 Jul 24 13:38:55.059549 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:55.059128 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerDied","Data":"888c19258cdb54cfe53bfdb792f442323dbe39eac7656ed0d197904f6567cd68"} Jul 24 13:38:56.853964 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:56.853926 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:56.853964 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:56.853953 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:56.854557 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:56.853968 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:56.854557 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:56.854055 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7j8ww" podUID="852819de-9209-4a8e-aa77-bf7f666a5dc1" Jul 24 13:38:56.854557 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:56.854209 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tjr2x" podUID="1ccb971b-88bb-4e91-aa7a-4006dca11af0" Jul 24 13:38:56.854557 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:56.854312 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnqvq" podUID="8b70c494-5fd3-4783-a114-907bf3f3805e" Jul 24 13:38:57.597359 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:57.597324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:57.597587 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:57.597374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:57.597587 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.597484 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:57.597587 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.597490 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:57.597587 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.597552 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret podName:8b70c494-5fd3-4783-a114-907bf3f3805e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:29.597534095 +0000 UTC m=+66.208013832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret") pod "global-pull-secret-syncer-vnqvq" (UID: "8b70c494-5fd3-4783-a114-907bf3f3805e") : object "kube-system"/"original-pull-secret" not registered Jul 24 13:38:57.597587 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.597574 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs podName:1ccb971b-88bb-4e91-aa7a-4006dca11af0 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:29.597564363 +0000 UTC m=+66.208044087 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs") pod "network-metrics-daemon-tjr2x" (UID: "1ccb971b-88bb-4e91-aa7a-4006dca11af0") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 13:38:57.698138 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:57.698105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:57.698320 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.698259 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 13:38:57.698320 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.698275 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 13:38:57.698320 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.698284 2568 projected.go:196] Error preparing data for projected volume kube-api-access-68gdz for pod openshift-network-diagnostics/network-check-target-7j8ww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:57.698476 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:57.698337 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz podName:852819de-9209-4a8e-aa77-bf7f666a5dc1 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:29.698320666 +0000 UTC m=+66.308800406 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-68gdz" (UniqueName: "kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz") pod "network-check-target-7j8ww" (UID: "852819de-9209-4a8e-aa77-bf7f666a5dc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 13:38:58.202852 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.202819 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-248.ec2.internal" event="NodeReady" Jul 24 13:38:58.203451 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.202968 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 13:38:58.238620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.238580 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b984f8d55-8xmml"] Jul 24 13:38:58.243129 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.243098 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.246354 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.246323 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 13:38:58.246517 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.246357 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l24vv\"" Jul 24 13:38:58.246517 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.246380 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 13:38:58.246794 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.246776 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 13:38:58.255577 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.255518 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld"] Jul 24 13:38:58.256034 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.255610 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 13:38:58.259509 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.259481 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn"] Jul 24 13:38:58.259641 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.259617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.263204 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263102 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 13:38:58.263204 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263131 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 13:38:58.263393 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263131 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 13:38:58.263393 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263211 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:38:58.263393 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263303 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qh8wr\"" Jul 24 13:38:58.263674 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263654 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls"] Jul 24 13:38:58.264180 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.263898 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:58.266876 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.266855 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5"] Jul 24 13:38:58.266989 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.266973 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.269224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.269203 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 13:38:58.269648 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.269628 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:38:58.270041 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.270024 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b984f8d55-8xmml"] Jul 24 13:38:58.270123 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.270057 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4g25d"] Jul 24 13:38:58.270183 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.270139 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" Jul 24 13:38:58.273483 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.273455 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.276674 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.276656 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cpc4j\"" Jul 24 13:38:58.276805 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.276784 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7xxqj\"" Jul 24 13:38:58.277032 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.277010 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 13:38:58.278106 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.278078 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 13:38:58.278194 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.278164 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 13:38:58.280102 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.280075 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 13:38:58.284966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.284540 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 13:38:58.285290 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.285267 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j9j4b\"" Jul 24 13:38:58.285393 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.285272 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 13:38:58.285393 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.285343 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld"] Jul 24 13:38:58.286926 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.286905 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 13:38:58.287020 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.286910 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 13:38:58.293765 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.293734 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-trkqq\"" Jul 24 13:38:58.294159 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.294132 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 13:38:58.297048 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.296942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5"] Jul 24 13:38:58.298195 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.298171 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls"] Jul 24 13:38:58.299111 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.298997 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4g25d"] Jul 24 13:38:58.324845 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.324816 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn"] Jul 24 13:38:58.370879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.370845 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k8lk6"] Jul 24 13:38:58.374228 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.374204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.376738 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.376688 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 13:38:58.376738 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.376712 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xl6tf\"" Jul 24 13:38:58.376923 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.376893 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 13:38:58.386935 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.386912 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k8lk6"] Jul 24 13:38:58.402193 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402157 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-image-registry-private-configuration\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.402378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-trusted-ca\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.402378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402267 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:58.402378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402312 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0108565-5b1f-4f9a-b7d5-f12319bf0358-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.402378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0108565-5b1f-4f9a-b7d5-f12319bf0358-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.402572 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrwb4\" (UniqueName: \"kubernetes.io/projected/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-kube-api-access-vrwb4\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.402572 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/883ffe42-65bc-4a6f-8f14-d4285bc94b90-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:58.402572 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-bound-sa-token\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.402572 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402528 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7dmd\" (UniqueName: \"kubernetes.io/projected/e0108565-5b1f-4f9a-b7d5-f12319bf0358-kube-api-access-x7dmd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.402572 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-certificates\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccglj\" (UniqueName: \"kubernetes.io/projected/ce1b7a05-d133-42c8-b60d-bd0708b7b452-kube-api-access-ccglj\") pod \"network-check-source-7d88cb55d8-x7pp5\" (UID: \"ce1b7a05-d133-42c8-b60d-bd0708b7b452\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-ca-trust-extracted\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40269b1-49be-404b-ab9e-c645775f0848-cert\") pod \"ingress-canary-4g25d\" (UID: \"a40269b1-49be-404b-ab9e-c645775f0848\") " pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402803 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-config\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.402855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqr5\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-kube-api-access-qxqr5\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.403132 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402967 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-installation-pull-secrets\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.403132 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.402997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxrg\" (UniqueName: \"kubernetes.io/projected/a40269b1-49be-404b-ab9e-c645775f0848-kube-api-access-6pxrg\") pod \"ingress-canary-4g25d\" (UID: \"a40269b1-49be-404b-ab9e-c645775f0848\") " pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.503642 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.503600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-ca-trust-extracted\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.503642 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.503651 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40269b1-49be-404b-ab9e-c645775f0848-cert\") pod \"ingress-canary-4g25d\" (UID: \"a40269b1-49be-404b-ab9e-c645775f0848\") " pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.503899 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.503674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.503899 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.503700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.503899 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.503741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-config\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.503899 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.503764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqr5\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-kube-api-access-qxqr5\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.503899 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.503839 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 13:38:58.503899 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.503858 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b984f8d55-8xmml: secret "image-registry-tls" not found Jul 24 13:38:58.504170 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.503934 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls podName:30ccbce5-4a45-4a1c-8b3f-ce069641e8e3 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:59.003912012 +0000 UTC m=+35.614391752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls") pod "image-registry-b984f8d55-8xmml" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3") : secret "image-registry-tls" not found Jul 24 13:38:58.504477 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.504396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-ca-trust-extracted\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.504680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80464ab-57c8-4be8-af66-31519ad6beaa-tmp-dir\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.504828 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-config\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.504960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-installation-pull-secrets\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.504999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxrg\" (UniqueName: \"kubernetes.io/projected/a40269b1-49be-404b-ab9e-c645775f0848-kube-api-access-6pxrg\") pod \"ingress-canary-4g25d\" (UID: \"a40269b1-49be-404b-ab9e-c645775f0848\") " pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-image-registry-private-configuration\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-trusted-ca\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0108565-5b1f-4f9a-b7d5-f12319bf0358-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0108565-5b1f-4f9a-b7d5-f12319bf0358-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505274 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrwb4\" (UniqueName: \"kubernetes.io/projected/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-kube-api-access-vrwb4\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d80464ab-57c8-4be8-af66-31519ad6beaa-config-volume\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/883ffe42-65bc-4a6f-8f14-d4285bc94b90-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505373 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqtf\" (UniqueName: \"kubernetes.io/projected/d80464ab-57c8-4be8-af66-31519ad6beaa-kube-api-access-6tqtf\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.507869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-bound-sa-token\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.508620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505461 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dmd\" (UniqueName: \"kubernetes.io/projected/e0108565-5b1f-4f9a-b7d5-f12319bf0358-kube-api-access-x7dmd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.508620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-certificates\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.508620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.505530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccglj\" (UniqueName: \"kubernetes.io/projected/ce1b7a05-d133-42c8-b60d-bd0708b7b452-kube-api-access-ccglj\") pod \"network-check-source-7d88cb55d8-x7pp5\" (UID: \"ce1b7a05-d133-42c8-b60d-bd0708b7b452\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" Jul 24 13:38:58.508620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.507274 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0108565-5b1f-4f9a-b7d5-f12319bf0358-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.508620 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.508018 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 13:38:58.508620 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.508093 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert podName:883ffe42-65bc-4a6f-8f14-d4285bc94b90 nodeName:}" failed. No retries permitted until 2026-07-24 13:38:59.008062498 +0000 UTC m=+35.618542234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bcjmn" (UID: "883ffe42-65bc-4a6f-8f14-d4285bc94b90") : secret "networking-console-plugin-cert" not found Jul 24 13:38:58.509669 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.509023 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-installation-pull-secrets\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.509669 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.509237 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.509669 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.509454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a40269b1-49be-404b-ab9e-c645775f0848-cert\") pod \"ingress-canary-4g25d\" (UID: \"a40269b1-49be-404b-ab9e-c645775f0848\") " pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.510159 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.510135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-certificates\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.510479 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.510318 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-trusted-ca\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.511117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.511095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/883ffe42-65bc-4a6f-8f14-d4285bc94b90-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:58.512154 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.512122 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0108565-5b1f-4f9a-b7d5-f12319bf0358-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.512154 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.512129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-image-registry-private-configuration\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.516670 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.516644 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqr5\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-kube-api-access-qxqr5\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.519319 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.519292 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-bound-sa-token\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:58.519528 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.519506 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrwb4\" (UniqueName: \"kubernetes.io/projected/afe86dfe-2fb9-46b8-a96c-876b4f8b1507-kube-api-access-vrwb4\") pod \"service-ca-operator-5d6c75f84f-wbpls\" (UID: \"afe86dfe-2fb9-46b8-a96c-876b4f8b1507\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.519610 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.519574 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dmd\" (UniqueName: \"kubernetes.io/projected/e0108565-5b1f-4f9a-b7d5-f12319bf0358-kube-api-access-x7dmd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld\" (UID: \"e0108565-5b1f-4f9a-b7d5-f12319bf0358\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.519754 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.519704 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccglj\" (UniqueName: \"kubernetes.io/projected/ce1b7a05-d133-42c8-b60d-bd0708b7b452-kube-api-access-ccglj\") pod \"network-check-source-7d88cb55d8-x7pp5\" (UID: \"ce1b7a05-d133-42c8-b60d-bd0708b7b452\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" Jul 24 13:38:58.521103 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.521083 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxrg\" (UniqueName: \"kubernetes.io/projected/a40269b1-49be-404b-ab9e-c645775f0848-kube-api-access-6pxrg\") pod \"ingress-canary-4g25d\" (UID: \"a40269b1-49be-404b-ab9e-c645775f0848\") " pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.572235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.572202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" Jul 24 13:38:58.587139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.587101 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" Jul 24 13:38:58.597644 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.597614 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" Jul 24 13:38:58.605114 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.605091 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4g25d" Jul 24 13:38:58.606402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.606382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.606483 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.606415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d80464ab-57c8-4be8-af66-31519ad6beaa-config-volume\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.606483 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.606447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqtf\" (UniqueName: \"kubernetes.io/projected/d80464ab-57c8-4be8-af66-31519ad6beaa-kube-api-access-6tqtf\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.606602 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.606510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80464ab-57c8-4be8-af66-31519ad6beaa-tmp-dir\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.606602 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.606547 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:38:58.606713 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:58.606620 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls podName:d80464ab-57c8-4be8-af66-31519ad6beaa nodeName:}" failed. No retries permitted until 2026-07-24 13:38:59.106596329 +0000 UTC m=+35.717076055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls") pod "dns-default-k8lk6" (UID: "d80464ab-57c8-4be8-af66-31519ad6beaa") : secret "dns-default-metrics-tls" not found Jul 24 13:38:58.607099 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.607077 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d80464ab-57c8-4be8-af66-31519ad6beaa-tmp-dir\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.607267 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.607249 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d80464ab-57c8-4be8-af66-31519ad6beaa-config-volume\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.615686 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.615664 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqtf\" (UniqueName: \"kubernetes.io/projected/d80464ab-57c8-4be8-af66-31519ad6beaa-kube-api-access-6tqtf\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:58.853486 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.853395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:38:58.853486 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.853427 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:38:58.853486 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.853441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:38:58.856804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.856777 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 13:38:58.856954 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.856832 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v4t2\"" Jul 24 13:38:58.857014 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.856973 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnhnl\"" Jul 24 13:38:58.857098 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:58.857068 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 13:38:59.010740 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.010695 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:38:59.010903 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.010856 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 13:38:59.010903 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.010878 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b984f8d55-8xmml: secret "image-registry-tls" not found Jul 24 13:38:59.011016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.010924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:38:59.011016 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.010938 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls podName:30ccbce5-4a45-4a1c-8b3f-ce069641e8e3 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:00.01091928 +0000 UTC m=+36.621399026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls") pod "image-registry-b984f8d55-8xmml" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3") : secret "image-registry-tls" not found Jul 24 13:38:59.011123 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.011074 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 13:38:59.011173 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.011150 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert podName:883ffe42-65bc-4a6f-8f14-d4285bc94b90 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:00.011131499 +0000 UTC m=+36.621611234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bcjmn" (UID: "883ffe42-65bc-4a6f-8f14-d4285bc94b90") : secret "networking-console-plugin-cert" not found Jul 24 13:38:59.111771 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.111654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:38:59.111953 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.111817 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:38:59.111953 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:38:59.111886 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls podName:d80464ab-57c8-4be8-af66-31519ad6beaa nodeName:}" failed. No retries permitted until 2026-07-24 13:39:00.111866494 +0000 UTC m=+36.722346236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls") pod "dns-default-k8lk6" (UID: "d80464ab-57c8-4be8-af66-31519ad6beaa") : secret "dns-default-metrics-tls" not found Jul 24 13:38:59.590850 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.590815 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls"] Jul 24 13:38:59.593540 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.593510 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4g25d"] Jul 24 13:38:59.594320 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.594264 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld"] Jul 24 13:38:59.595261 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:59.595236 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe86dfe_2fb9_46b8_a96c_876b4f8b1507.slice/crio-b6c8a9ef376b9282c92276ad16333ba2e102feb869ea3f9f17334eced5bde670 WatchSource:0}: Error finding container b6c8a9ef376b9282c92276ad16333ba2e102feb869ea3f9f17334eced5bde670: Status 404 returned error can't find the container with id b6c8a9ef376b9282c92276ad16333ba2e102feb869ea3f9f17334eced5bde670 Jul 24 13:38:59.597310 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:59.597241 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0108565_5b1f_4f9a_b7d5_f12319bf0358.slice/crio-a0c7bab5ea421ad13f9fb65da1bd666960384ab17b1c010ab9eaad1c6c6e47e5 WatchSource:0}: Error finding container a0c7bab5ea421ad13f9fb65da1bd666960384ab17b1c010ab9eaad1c6c6e47e5: Status 404 returned error can't find the container with id a0c7bab5ea421ad13f9fb65da1bd666960384ab17b1c010ab9eaad1c6c6e47e5 Jul 24 13:38:59.598197 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:59.598173 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40269b1_49be_404b_ab9e_c645775f0848.slice/crio-0d165b6ca1e1f895384248bc88f482805b0e48eafe9f8a3377e809ea6beeee6e WatchSource:0}: Error finding container 0d165b6ca1e1f895384248bc88f482805b0e48eafe9f8a3377e809ea6beeee6e: Status 404 returned error can't find the container with id 0d165b6ca1e1f895384248bc88f482805b0e48eafe9f8a3377e809ea6beeee6e Jul 24 13:38:59.630399 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:38:59.620110 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5"] Jul 24 13:38:59.635412 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:38:59.635353 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1b7a05_d133_42c8_b60d_bd0708b7b452.slice/crio-b9e2e804ef46de4299caea4b2d8c4b33117935dabcc13d65adc120b5ed4acde0 WatchSource:0}: Error finding container b9e2e804ef46de4299caea4b2d8c4b33117935dabcc13d65adc120b5ed4acde0: Status 404 returned error can't find the container with id b9e2e804ef46de4299caea4b2d8c4b33117935dabcc13d65adc120b5ed4acde0 Jul 24 13:39:00.023685 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.023589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:00.023685 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.023669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:00.023927 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.023770 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 13:39:00.023927 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.023792 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b984f8d55-8xmml: secret "image-registry-tls" not found Jul 24 13:39:00.023927 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.023803 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 13:39:00.023927 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.023854 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls podName:30ccbce5-4a45-4a1c-8b3f-ce069641e8e3 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:02.023838049 +0000 UTC m=+38.634317773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls") pod "image-registry-b984f8d55-8xmml" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3") : secret "image-registry-tls" not found Jul 24 13:39:00.023927 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.023868 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert podName:883ffe42-65bc-4a6f-8f14-d4285bc94b90 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:02.023861681 +0000 UTC m=+38.634341405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bcjmn" (UID: "883ffe42-65bc-4a6f-8f14-d4285bc94b90") : secret "networking-console-plugin-cert" not found Jul 24 13:39:00.075030 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.074999 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef5c0207-b134-4604-8b0e-14ea1481ee07" containerID="fb29389558df2dd1a6fb2ab148fb853955f576fee069c51ed6b474d65d1a947b" exitCode=0 Jul 24 13:39:00.075181 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.075080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerDied","Data":"fb29389558df2dd1a6fb2ab148fb853955f576fee069c51ed6b474d65d1a947b"} Jul 24 13:39:00.076286 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.076262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" event={"ID":"e0108565-5b1f-4f9a-b7d5-f12319bf0358","Type":"ContainerStarted","Data":"a0c7bab5ea421ad13f9fb65da1bd666960384ab17b1c010ab9eaad1c6c6e47e5"} Jul 24 13:39:00.077352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.077325 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4g25d" event={"ID":"a40269b1-49be-404b-ab9e-c645775f0848","Type":"ContainerStarted","Data":"0d165b6ca1e1f895384248bc88f482805b0e48eafe9f8a3377e809ea6beeee6e"} Jul 24 13:39:00.078473 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.078453 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" event={"ID":"afe86dfe-2fb9-46b8-a96c-876b4f8b1507","Type":"ContainerStarted","Data":"b6c8a9ef376b9282c92276ad16333ba2e102feb869ea3f9f17334eced5bde670"} Jul 24 13:39:00.079628 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.079608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" event={"ID":"ce1b7a05-d133-42c8-b60d-bd0708b7b452","Type":"ContainerStarted","Data":"b9e2e804ef46de4299caea4b2d8c4b33117935dabcc13d65adc120b5ed4acde0"} Jul 24 13:39:00.124407 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:00.124380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:00.124519 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.124496 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:00.124555 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:00.124544 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls podName:d80464ab-57c8-4be8-af66-31519ad6beaa nodeName:}" failed. No retries permitted until 2026-07-24 13:39:02.124531567 +0000 UTC m=+38.735011291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls") pod "dns-default-k8lk6" (UID: "d80464ab-57c8-4be8-af66-31519ad6beaa") : secret "dns-default-metrics-tls" not found Jul 24 13:39:01.099374 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:01.099280 2568 generic.go:356] "Generic (PLEG): container finished" podID="ef5c0207-b134-4604-8b0e-14ea1481ee07" containerID="2ebbac792bb2bcc3b4c69244d72963b8855d1d4bf1c2189b5513f91925bffa42" exitCode=0 Jul 24 13:39:01.099374 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:01.099367 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerDied","Data":"2ebbac792bb2bcc3b4c69244d72963b8855d1d4bf1c2189b5513f91925bffa42"} Jul 24 13:39:02.042629 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:02.042375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:02.042831 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.042558 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 13:39:02.042831 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.042763 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b984f8d55-8xmml: secret "image-registry-tls" not found Jul 24 13:39:02.042831 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:02.042716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:02.042831 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.042819 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 13:39:02.042831 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.042836 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls podName:30ccbce5-4a45-4a1c-8b3f-ce069641e8e3 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:06.042814592 +0000 UTC m=+42.653294339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls") pod "image-registry-b984f8d55-8xmml" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3") : secret "image-registry-tls" not found Jul 24 13:39:02.043037 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.042899 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert podName:883ffe42-65bc-4a6f-8f14-d4285bc94b90 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:06.04287645 +0000 UTC m=+42.653356189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bcjmn" (UID: "883ffe42-65bc-4a6f-8f14-d4285bc94b90") : secret "networking-console-plugin-cert" not found Jul 24 13:39:02.105168 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:02.105136 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" event={"ID":"ef5c0207-b134-4604-8b0e-14ea1481ee07","Type":"ContainerStarted","Data":"b663b5f55a4b3cc63434e55e109ee84d47daf9db6a2197d8e1271b1d8da0e291"} Jul 24 13:39:02.131480 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:02.131426 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wtfp4" podStartSLOduration=8.110981821 podStartE2EDuration="38.131409608s" totalImagesPullingTime="30.020427787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:02.129686023 +0000 UTC m=+38.740165770" watchObservedRunningTime="2026-07-24 13:39:02.131409608 +0000 UTC m=+38.741889353" Jul 24 13:39:02.144031 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:02.143992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:02.144189 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.144155 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:02.144243 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:02.144230 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls podName:d80464ab-57c8-4be8-af66-31519ad6beaa nodeName:}" failed. No retries permitted until 2026-07-24 13:39:06.144211088 +0000 UTC m=+42.754690829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls") pod "dns-default-k8lk6" (UID: "d80464ab-57c8-4be8-af66-31519ad6beaa") : secret "dns-default-metrics-tls" not found Jul 24 13:39:06.077783 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.077707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:06.078307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.077829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:06.078307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.077893 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 13:39:06.078307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.077919 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b984f8d55-8xmml: secret "image-registry-tls" not found Jul 24 13:39:06.078307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.077956 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 13:39:06.078307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.077983 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls podName:30ccbce5-4a45-4a1c-8b3f-ce069641e8e3 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:14.077961771 +0000 UTC m=+50.688441506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls") pod "image-registry-b984f8d55-8xmml" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3") : secret "image-registry-tls" not found Jul 24 13:39:06.078307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.078006 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert podName:883ffe42-65bc-4a6f-8f14-d4285bc94b90 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:14.077991478 +0000 UTC m=+50.688471214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bcjmn" (UID: "883ffe42-65bc-4a6f-8f14-d4285bc94b90") : secret "networking-console-plugin-cert" not found Jul 24 13:39:06.117243 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.117203 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" event={"ID":"afe86dfe-2fb9-46b8-a96c-876b4f8b1507","Type":"ContainerStarted","Data":"0db3c2ea0f4099806834127ceb9d72702dc369f886be84ed941e2aa8509c465a"} Jul 24 13:39:06.119151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.119111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" event={"ID":"ce1b7a05-d133-42c8-b60d-bd0708b7b452","Type":"ContainerStarted","Data":"b074f025ba69d2b4484eff6cb78c71387e7fb9194cf58210302545766bae5088"} Jul 24 13:39:06.120795 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.120761 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" event={"ID":"e0108565-5b1f-4f9a-b7d5-f12319bf0358","Type":"ContainerStarted","Data":"5bae1e21e7c3cd27e2a606815b177c93ab67c60d91fa18f9752daa12ee94bf42"} Jul 24 13:39:06.122678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.122650 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4g25d" event={"ID":"a40269b1-49be-404b-ab9e-c645775f0848","Type":"ContainerStarted","Data":"2690e8151aa59c2e24f0226575743b640ce875608ba9ef8941038719e0d67eec"} Jul 24 13:39:06.135097 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.135037 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" podStartSLOduration=32.569319983 podStartE2EDuration="38.135019262s" totalImagesPullingTime="5.565699279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:06.133957085 +0000 UTC m=+42.744436815" watchObservedRunningTime="2026-07-24 13:39:06.135019262 +0000 UTC m=+42.745499010" Jul 24 13:39:06.150747 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.150667 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4g25d" podStartSLOduration=2.592478035 podStartE2EDuration="8.150638007s" totalImagesPullingTime="5.558159972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:06.150312712 +0000 UTC m=+42.760792458" watchObservedRunningTime="2026-07-24 13:39:06.150638007 +0000 UTC m=+42.761117751" Jul 24 13:39:06.166897 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.166507 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" podStartSLOduration=34.602972503 podStartE2EDuration="40.16648872s" totalImagesPullingTime="5.563516217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:06.165388357 +0000 UTC m=+42.775868095" watchObservedRunningTime="2026-07-24 13:39:06.16648872 +0000 UTC m=+42.776968467" Jul 24 13:39:06.178747 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.178693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:06.178912 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.178850 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:06.178912 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:06.178909 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls podName:d80464ab-57c8-4be8-af66-31519ad6beaa nodeName:}" failed. No retries permitted until 2026-07-24 13:39:14.17889483 +0000 UTC m=+50.789374554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls") pod "dns-default-k8lk6" (UID: "d80464ab-57c8-4be8-af66-31519ad6beaa") : secret "dns-default-metrics-tls" not found Jul 24 13:39:06.183964 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:06.183902 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x7pp5" podStartSLOduration=31.647969023 podStartE2EDuration="37.183884902s" totalImagesPullingTime="5.535915879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:06.183101525 +0000 UTC m=+42.793581271" watchObservedRunningTime="2026-07-24 13:39:06.183884902 +0000 UTC m=+42.794364647" Jul 24 13:39:14.146028 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:14.145983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:14.146580 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:14.146063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:14.146580 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.146151 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 13:39:14.146580 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.146167 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 13:39:14.146580 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.146190 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b984f8d55-8xmml: secret "image-registry-tls" not found Jul 24 13:39:14.146580 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.146212 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert podName:883ffe42-65bc-4a6f-8f14-d4285bc94b90 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:30.146198668 +0000 UTC m=+66.756678392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bcjmn" (UID: "883ffe42-65bc-4a6f-8f14-d4285bc94b90") : secret "networking-console-plugin-cert" not found Jul 24 13:39:14.146580 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.146260 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls podName:30ccbce5-4a45-4a1c-8b3f-ce069641e8e3 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:30.146241568 +0000 UTC m=+66.756721309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls") pod "image-registry-b984f8d55-8xmml" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3") : secret "image-registry-tls" not found Jul 24 13:39:14.247193 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:14.247154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:14.247367 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.247254 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 13:39:14.247367 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:14.247329 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls podName:d80464ab-57c8-4be8-af66-31519ad6beaa nodeName:}" failed. No retries permitted until 2026-07-24 13:39:30.247314234 +0000 UTC m=+66.857793958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls") pod "dns-default-k8lk6" (UID: "d80464ab-57c8-4be8-af66-31519ad6beaa") : secret "dns-default-metrics-tls" not found Jul 24 13:39:24.069005 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:24.068975 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2llgg" Jul 24 13:39:29.673159 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.673115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:39:29.673159 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.673155 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:39:29.675846 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.675823 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 13:39:29.676697 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.676679 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 13:39:29.687621 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.687586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8b70c494-5fd3-4783-a114-907bf3f3805e-original-pull-secret\") pod \"global-pull-secret-syncer-vnqvq\" (UID: \"8b70c494-5fd3-4783-a114-907bf3f3805e\") " pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:39:29.687769 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.687651 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ccb971b-88bb-4e91-aa7a-4006dca11af0-metrics-certs\") pod \"network-metrics-daemon-tjr2x\" (UID: \"1ccb971b-88bb-4e91-aa7a-4006dca11af0\") " pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:39:29.773650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.773603 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnqvq" Jul 24 13:39:29.773847 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.773717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:39:29.776153 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.776128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gdz\" (UniqueName: \"kubernetes.io/projected/852819de-9209-4a8e-aa77-bf7f666a5dc1-kube-api-access-68gdz\") pod \"network-check-target-7j8ww\" (UID: \"852819de-9209-4a8e-aa77-bf7f666a5dc1\") " pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:39:29.782069 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.782046 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v4t2\"" Jul 24 13:39:29.790121 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.790083 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tjr2x" Jul 24 13:39:29.911536 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.911502 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnqvq"] Jul 24 13:39:29.915753 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:29.915689 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b70c494_5fd3_4783_a114_907bf3f3805e.slice/crio-c4c44d8b79df7f7e672fb72a0e96992a307739b45952ed35abac10e6c9d4b933 WatchSource:0}: Error finding container c4c44d8b79df7f7e672fb72a0e96992a307739b45952ed35abac10e6c9d4b933: Status 404 returned error can't find the container with id c4c44d8b79df7f7e672fb72a0e96992a307739b45952ed35abac10e6c9d4b933 Jul 24 13:39:29.927975 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:29.927916 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tjr2x"] Jul 24 13:39:29.930568 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:29.930543 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ccb971b_88bb_4e91_aa7a_4006dca11af0.slice/crio-eec582f00c26fad3e7e6709d9608efd71c32ab1d7aac50d57af35dc1c5e8f15f WatchSource:0}: Error finding container eec582f00c26fad3e7e6709d9608efd71c32ab1d7aac50d57af35dc1c5e8f15f: Status 404 returned error can't find the container with id eec582f00c26fad3e7e6709d9608efd71c32ab1d7aac50d57af35dc1c5e8f15f Jul 24 13:39:30.068643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.068607 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bnhnl\"" Jul 24 13:39:30.076104 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.076082 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:39:30.177164 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.177130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:30.177336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.177213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:30.179593 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.179526 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"image-registry-b984f8d55-8xmml\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:30.179696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.179665 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/883ffe42-65bc-4a6f-8f14-d4285bc94b90-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bcjmn\" (UID: \"883ffe42-65bc-4a6f-8f14-d4285bc94b90\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:30.185243 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.185213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjr2x" event={"ID":"1ccb971b-88bb-4e91-aa7a-4006dca11af0","Type":"ContainerStarted","Data":"eec582f00c26fad3e7e6709d9608efd71c32ab1d7aac50d57af35dc1c5e8f15f"} Jul 24 13:39:30.186474 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.186450 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnqvq" event={"ID":"8b70c494-5fd3-4783-a114-907bf3f3805e","Type":"ContainerStarted","Data":"c4c44d8b79df7f7e672fb72a0e96992a307739b45952ed35abac10e6c9d4b933"} Jul 24 13:39:30.207432 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.207403 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7j8ww"] Jul 24 13:39:30.211548 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:30.211524 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852819de_9209_4a8e_aa77_bf7f666a5dc1.slice/crio-96b5766b8a90b2d71a5f2d815498d2aa66a9af75044c7204a9fed0e4b4fa8237 WatchSource:0}: Error finding container 96b5766b8a90b2d71a5f2d815498d2aa66a9af75044c7204a9fed0e4b4fa8237: Status 404 returned error can't find the container with id 96b5766b8a90b2d71a5f2d815498d2aa66a9af75044c7204a9fed0e4b4fa8237 Jul 24 13:39:30.277578 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.277546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:30.279806 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.279789 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d80464ab-57c8-4be8-af66-31519ad6beaa-metrics-tls\") pod \"dns-default-k8lk6\" (UID: \"d80464ab-57c8-4be8-af66-31519ad6beaa\") " pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:30.358065 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.358033 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l24vv\"" Jul 24 13:39:30.366165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.366147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:30.382535 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.382506 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cpc4j\"" Jul 24 13:39:30.389835 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.389819 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" Jul 24 13:39:30.487944 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.487306 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xl6tf\"" Jul 24 13:39:30.495121 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.495094 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:30.539558 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.539528 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b984f8d55-8xmml"] Jul 24 13:39:30.544906 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:30.544862 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ccbce5_4a45_4a1c_8b3f_ce069641e8e3.slice/crio-425275734bfc016e182b6412f2186081dec34e31b16566ed4cef095ec5081a3a WatchSource:0}: Error finding container 425275734bfc016e182b6412f2186081dec34e31b16566ed4cef095ec5081a3a: Status 404 returned error can't find the container with id 425275734bfc016e182b6412f2186081dec34e31b16566ed4cef095ec5081a3a Jul 24 13:39:30.557014 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.556980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn"] Jul 24 13:39:30.562113 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:30.562079 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883ffe42_65bc_4a6f_8f14_d4285bc94b90.slice/crio-0f50aadf1fd7daaf6aa484c8a751f5a1b3f3ed99ebc8405d8b7d42da563899ea WatchSource:0}: Error finding container 0f50aadf1fd7daaf6aa484c8a751f5a1b3f3ed99ebc8405d8b7d42da563899ea: Status 404 returned error can't find the container with id 0f50aadf1fd7daaf6aa484c8a751f5a1b3f3ed99ebc8405d8b7d42da563899ea Jul 24 13:39:30.682989 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:30.682956 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k8lk6"] Jul 24 13:39:30.690609 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:30.687369 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80464ab_57c8_4be8_af66_31519ad6beaa.slice/crio-c209d5863a9135d61e63bed7d2cdafce2a141e9e589880ddeb4f2175e2e15c9c WatchSource:0}: Error finding container c209d5863a9135d61e63bed7d2cdafce2a141e9e589880ddeb4f2175e2e15c9c: Status 404 returned error can't find the container with id c209d5863a9135d61e63bed7d2cdafce2a141e9e589880ddeb4f2175e2e15c9c Jul 24 13:39:31.191783 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.191693 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k8lk6" event={"ID":"d80464ab-57c8-4be8-af66-31519ad6beaa","Type":"ContainerStarted","Data":"c209d5863a9135d61e63bed7d2cdafce2a141e9e589880ddeb4f2175e2e15c9c"} Jul 24 13:39:31.193145 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.193101 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" event={"ID":"883ffe42-65bc-4a6f-8f14-d4285bc94b90","Type":"ContainerStarted","Data":"0f50aadf1fd7daaf6aa484c8a751f5a1b3f3ed99ebc8405d8b7d42da563899ea"} Jul 24 13:39:31.194870 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.194839 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" event={"ID":"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3","Type":"ContainerStarted","Data":"22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa"} Jul 24 13:39:31.195008 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.194893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" event={"ID":"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3","Type":"ContainerStarted","Data":"425275734bfc016e182b6412f2186081dec34e31b16566ed4cef095ec5081a3a"} Jul 24 13:39:31.195008 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.194934 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:31.196721 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.196691 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7j8ww" event={"ID":"852819de-9209-4a8e-aa77-bf7f666a5dc1","Type":"ContainerStarted","Data":"cd0f12ecf7a347b15decc4e9306e53b77f645e14f0e5604f80e174eeb8e37ee1"} Jul 24 13:39:31.196861 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.196749 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7j8ww" event={"ID":"852819de-9209-4a8e-aa77-bf7f666a5dc1","Type":"ContainerStarted","Data":"96b5766b8a90b2d71a5f2d815498d2aa66a9af75044c7204a9fed0e4b4fa8237"} Jul 24 13:39:31.196861 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.196829 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:39:31.216805 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.216742 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" podStartSLOduration=67.216696013 podStartE2EDuration="1m7.216696013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:31.215441895 +0000 UTC m=+67.825921641" watchObservedRunningTime="2026-07-24 13:39:31.216696013 +0000 UTC m=+67.827175760" Jul 24 13:39:31.230042 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.229985 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7j8ww" podStartSLOduration=67.229962566 podStartE2EDuration="1m7.229962566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:31.229482496 +0000 UTC m=+67.839962243" watchObservedRunningTime="2026-07-24 13:39:31.229962566 +0000 UTC m=+67.840442316" Jul 24 13:39:31.705829 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.705627 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b984f8d55-8xmml"] Jul 24 13:39:31.774703 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.774547 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv"] Jul 24 13:39:31.803073 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.802932 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-cqct5"] Jul 24 13:39:31.803465 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.803437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" Jul 24 13:39:31.808017 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.807348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-q9lrl\"" Jul 24 13:39:31.808017 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.807652 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 13:39:31.808017 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.807876 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:39:31.819862 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.816845 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-fhj28"] Jul 24 13:39:31.819862 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.818175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.821828 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.821374 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 13:39:31.821828 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.821630 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qv97c\"" Jul 24 13:39:31.822469 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.822448 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 13:39:31.824677 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.824289 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:39:31.824677 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.824492 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 13:39:31.827810 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.827790 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 13:39:31.837660 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.837225 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx"] Jul 24 13:39:31.838961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.837924 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.840710 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.840632 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 13:39:31.841471 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.841451 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pbzds\"" Jul 24 13:39:31.841747 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.841710 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 13:39:31.842928 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.842906 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 13:39:31.843281 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.843126 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 13:39:31.849432 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.849413 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 13:39:31.861215 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.861178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv"] Jul 24 13:39:31.861455 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.861439 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:31.866751 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.866427 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 13:39:31.866751 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.866448 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-b9nqv\"" Jul 24 13:39:31.866944 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.866830 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 13:39:31.866944 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.866846 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 13:39:31.867459 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.867199 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 13:39:31.870162 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.870141 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx"] Jul 24 13:39:31.870278 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.870222 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-cqct5"] Jul 24 13:39:31.870278 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.870250 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-fhj28"] Jul 24 13:39:31.875228 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.875206 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fc58d8d96-zhw59"] Jul 24 13:39:31.889908 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.889873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500cd154-214f-47e3-b02e-e5bdeeef3f45-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.890078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.889941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/500cd154-214f-47e3-b02e-e5bdeeef3f45-tmp\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.890078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.889969 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2254edb8-c574-45db-99a8-320568e891cc-trusted-ca\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.890078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890013 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2254edb8-c574-45db-99a8-320568e891cc-config\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.890078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rl2r\" (UniqueName: \"kubernetes.io/projected/2254edb8-c574-45db-99a8-320568e891cc-kube-api-access-9rl2r\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.890078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/500cd154-214f-47e3-b02e-e5bdeeef3f45-snapshots\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.890338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/500cd154-214f-47e3-b02e-e5bdeeef3f45-serving-cert\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.890338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2254edb8-c574-45db-99a8-320568e891cc-serving-cert\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.890338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890155 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7l9m\" (UniqueName: \"kubernetes.io/projected/22bf929a-f86e-40a9-bf36-2a974ed98596-kube-api-access-k7l9m\") pod \"volume-data-source-validator-549c6d54cf-25qsv\" (UID: \"22bf929a-f86e-40a9-bf36-2a974ed98596\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" Jul 24 13:39:31.890338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqx5\" (UniqueName: \"kubernetes.io/projected/500cd154-214f-47e3-b02e-e5bdeeef3f45-kube-api-access-7hqx5\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.890338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.890210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500cd154-214f-47e3-b02e-e5bdeeef3f45-service-ca-bundle\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.896033 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.895939 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fc58d8d96-zhw59"] Jul 24 13:39:31.896167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.896073 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991107 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7l9m\" (UniqueName: \"kubernetes.io/projected/22bf929a-f86e-40a9-bf36-2a974ed98596-kube-api-access-k7l9m\") pod \"volume-data-source-validator-549c6d54cf-25qsv\" (UID: \"22bf929a-f86e-40a9-bf36-2a974ed98596\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" Jul 24 13:39:31.991107 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqx5\" (UniqueName: \"kubernetes.io/projected/500cd154-214f-47e3-b02e-e5bdeeef3f45-kube-api-access-7hqx5\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.991107 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/287e3ad6-10ce-4881-96cd-a179071bacb9-installation-pull-secrets\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500cd154-214f-47e3-b02e-e5bdeeef3f45-service-ca-bundle\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/287e3ad6-10ce-4881-96cd-a179071bacb9-trusted-ca\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500cd154-214f-47e3-b02e-e5bdeeef3f45-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991219 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/287e3ad6-10ce-4881-96cd-a179071bacb9-image-registry-private-configuration\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/287e3ad6-10ce-4881-96cd-a179071bacb9-ca-trust-extracted\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/287e3ad6-10ce-4881-96cd-a179071bacb9-registry-certificates\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a30465-6733-4640-b754-da43d4c2c0b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j56z\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-kube-api-access-9j56z\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.991394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991387 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/500cd154-214f-47e3-b02e-e5bdeeef3f45-tmp\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.991859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2254edb8-c574-45db-99a8-320568e891cc-trusted-ca\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.991859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.991463 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-bound-sa-token\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.992442 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.992416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2254edb8-c574-45db-99a8-320568e891cc-trusted-ca\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.992536 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.992506 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500cd154-214f-47e3-b02e-e5bdeeef3f45-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.992594 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.992567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-registry-tls\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:31.992657 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.992613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/49a30465-6733-4640-b754-da43d4c2c0b2-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:31.992711 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.992693 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/500cd154-214f-47e3-b02e-e5bdeeef3f45-tmp\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.993097 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2254edb8-c574-45db-99a8-320568e891cc-config\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.993184 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993169 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/500cd154-214f-47e3-b02e-e5bdeeef3f45-service-ca-bundle\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.993247 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rl2r\" (UniqueName: \"kubernetes.io/projected/2254edb8-c574-45db-99a8-320568e891cc-kube-api-access-9rl2r\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.993299 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr4wb\" (UniqueName: \"kubernetes.io/projected/49a30465-6733-4640-b754-da43d4c2c0b2-kube-api-access-nr4wb\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:31.993299 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/500cd154-214f-47e3-b02e-e5bdeeef3f45-snapshots\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.993397 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/500cd154-214f-47e3-b02e-e5bdeeef3f45-serving-cert\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.993397 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993343 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2254edb8-c574-45db-99a8-320568e891cc-serving-cert\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.993636 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.993617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2254edb8-c574-45db-99a8-320568e891cc-config\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:31.994346 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.994316 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/500cd154-214f-47e3-b02e-e5bdeeef3f45-snapshots\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.997444 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.997402 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/500cd154-214f-47e3-b02e-e5bdeeef3f45-serving-cert\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:31.998040 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:31.997996 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2254edb8-c574-45db-99a8-320568e891cc-serving-cert\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:32.004122 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.004092 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqx5\" (UniqueName: \"kubernetes.io/projected/500cd154-214f-47e3-b02e-e5bdeeef3f45-kube-api-access-7hqx5\") pod \"insights-operator-5885fdb759-fhj28\" (UID: \"500cd154-214f-47e3-b02e-e5bdeeef3f45\") " pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:32.005397 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.005350 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7l9m\" (UniqueName: \"kubernetes.io/projected/22bf929a-f86e-40a9-bf36-2a974ed98596-kube-api-access-k7l9m\") pod \"volume-data-source-validator-549c6d54cf-25qsv\" (UID: \"22bf929a-f86e-40a9-bf36-2a974ed98596\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" Jul 24 13:39:32.006071 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.006050 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rl2r\" (UniqueName: \"kubernetes.io/projected/2254edb8-c574-45db-99a8-320568e891cc-kube-api-access-9rl2r\") pod \"console-operator-584dcb6b54-cqct5\" (UID: \"2254edb8-c574-45db-99a8-320568e891cc\") " pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:32.094280 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/287e3ad6-10ce-4881-96cd-a179071bacb9-ca-trust-extracted\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094291 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/287e3ad6-10ce-4881-96cd-a179071bacb9-registry-certificates\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a30465-6733-4640-b754-da43d4c2c0b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.094450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j56z\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-kube-api-access-9j56z\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-bound-sa-token\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-registry-tls\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/49a30465-6733-4640-b754-da43d4c2c0b2-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.094720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr4wb\" (UniqueName: \"kubernetes.io/projected/49a30465-6733-4640-b754-da43d4c2c0b2-kube-api-access-nr4wb\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.094720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094559 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/287e3ad6-10ce-4881-96cd-a179071bacb9-installation-pull-secrets\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/287e3ad6-10ce-4881-96cd-a179071bacb9-trusted-ca\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.094720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.094612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/287e3ad6-10ce-4881-96cd-a179071bacb9-image-registry-private-configuration\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.095935 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.095906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/287e3ad6-10ce-4881-96cd-a179071bacb9-ca-trust-extracted\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.096942 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.096582 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/287e3ad6-10ce-4881-96cd-a179071bacb9-registry-certificates\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.097945 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.097440 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/49a30465-6733-4640-b754-da43d4c2c0b2-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.097945 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.097904 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/287e3ad6-10ce-4881-96cd-a179071bacb9-trusted-ca\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.098560 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.098510 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/287e3ad6-10ce-4881-96cd-a179071bacb9-image-registry-private-configuration\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.099497 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.099454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-registry-tls\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.100352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.100326 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/287e3ad6-10ce-4881-96cd-a179071bacb9-installation-pull-secrets\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.102205 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.101585 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/49a30465-6733-4640-b754-da43d4c2c0b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.104893 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.104868 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j56z\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-kube-api-access-9j56z\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.106138 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.106097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr4wb\" (UniqueName: \"kubernetes.io/projected/49a30465-6733-4640-b754-da43d4c2c0b2-kube-api-access-nr4wb\") pod \"cluster-monitoring-operator-548b489cb8-kqlvx\" (UID: \"49a30465-6733-4640-b754-da43d4c2c0b2\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.107255 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.107214 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/287e3ad6-10ce-4881-96cd-a179071bacb9-bound-sa-token\") pod \"image-registry-5fc58d8d96-zhw59\" (UID: \"287e3ad6-10ce-4881-96cd-a179071bacb9\") " pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.125156 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.125123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" Jul 24 13:39:32.134340 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.134306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:32.154548 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.154456 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-fhj28" Jul 24 13:39:32.176116 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.176079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" Jul 24 13:39:32.205083 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.205048 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjr2x" event={"ID":"1ccb971b-88bb-4e91-aa7a-4006dca11af0","Type":"ContainerStarted","Data":"ab2183778fe169ecca2a882864764683b33b26fe0702cb42087f9646c828fd30"} Jul 24 13:39:32.205237 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.205135 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tjr2x" event={"ID":"1ccb971b-88bb-4e91-aa7a-4006dca11af0","Type":"ContainerStarted","Data":"e201a5aadb05dcda4d622627acff98c4e36c2b93e446a7b0c261c78c285116df"} Jul 24 13:39:32.214317 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.214289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:32.222344 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.221924 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tjr2x" podStartSLOduration=66.667851041 podStartE2EDuration="1m8.221903225s" totalImagesPullingTime="1.554052184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:32.220327246 +0000 UTC m=+68.830806992" watchObservedRunningTime="2026-07-24 13:39:32.221903225 +0000 UTC m=+68.832382972" Jul 24 13:39:32.742297 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.742042 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-fhj28"] Jul 24 13:39:32.750501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.750472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx"] Jul 24 13:39:32.897556 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:32.897520 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500cd154_214f_47e3_b02e_e5bdeeef3f45.slice/crio-0ec6a6d5265a251e3846d56a77c523c899532aa446f1e63c701a3c589a56aaa4 WatchSource:0}: Error finding container 0ec6a6d5265a251e3846d56a77c523c899532aa446f1e63c701a3c589a56aaa4: Status 404 returned error can't find the container with id 0ec6a6d5265a251e3846d56a77c523c899532aa446f1e63c701a3c589a56aaa4 Jul 24 13:39:32.898182 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:32.898077 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a30465_6733_4640_b754_da43d4c2c0b2.slice/crio-1a90ed0998ed1289b609f7d0e2f17e9fc8105e1e3f51f88e34034322dd5e1fab WatchSource:0}: Error finding container 1a90ed0998ed1289b609f7d0e2f17e9fc8105e1e3f51f88e34034322dd5e1fab: Status 404 returned error can't find the container with id 1a90ed0998ed1289b609f7d0e2f17e9fc8105e1e3f51f88e34034322dd5e1fab Jul 24 13:39:32.982972 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.982907 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv"] Jul 24 13:39:32.985501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.985476 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-cqct5"] Jul 24 13:39:32.989385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:32.989360 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fc58d8d96-zhw59"] Jul 24 13:39:33.210590 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:33.210483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-fhj28" event={"ID":"500cd154-214f-47e3-b02e-e5bdeeef3f45","Type":"ContainerStarted","Data":"0ec6a6d5265a251e3846d56a77c523c899532aa446f1e63c701a3c589a56aaa4"} Jul 24 13:39:33.212167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:33.212134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" event={"ID":"49a30465-6733-4640-b754-da43d4c2c0b2","Type":"ContainerStarted","Data":"1a90ed0998ed1289b609f7d0e2f17e9fc8105e1e3f51f88e34034322dd5e1fab"} Jul 24 13:39:34.890962 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:34.890919 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bf929a_f86e_40a9_bf36_2a974ed98596.slice/crio-13024cfd09da5390041637c8e6894aa2c125f5837c967e66347cf76e1bbb0ada WatchSource:0}: Error finding container 13024cfd09da5390041637c8e6894aa2c125f5837c967e66347cf76e1bbb0ada: Status 404 returned error can't find the container with id 13024cfd09da5390041637c8e6894aa2c125f5837c967e66347cf76e1bbb0ada Jul 24 13:39:34.891385 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:34.891345 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2254edb8_c574_45db_99a8_320568e891cc.slice/crio-f7183e8c81cecebda5119c297579f0032e269fcc875476f87ac9aa8d622d1435 WatchSource:0}: Error finding container f7183e8c81cecebda5119c297579f0032e269fcc875476f87ac9aa8d622d1435: Status 404 returned error can't find the container with id f7183e8c81cecebda5119c297579f0032e269fcc875476f87ac9aa8d622d1435 Jul 24 13:39:34.892283 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:34.892253 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287e3ad6_10ce_4881_96cd_a179071bacb9.slice/crio-8f43193ab7c19d7239bad174ec4d618994850938e1e8dd605e3e6f04e3c44264 WatchSource:0}: Error finding container 8f43193ab7c19d7239bad174ec4d618994850938e1e8dd605e3e6f04e3c44264: Status 404 returned error can't find the container with id 8f43193ab7c19d7239bad174ec4d618994850938e1e8dd605e3e6f04e3c44264 Jul 24 13:39:35.223815 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:35.223641 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" event={"ID":"22bf929a-f86e-40a9-bf36-2a974ed98596","Type":"ContainerStarted","Data":"13024cfd09da5390041637c8e6894aa2c125f5837c967e66347cf76e1bbb0ada"} Jul 24 13:39:35.238406 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:35.238311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" event={"ID":"287e3ad6-10ce-4881-96cd-a179071bacb9","Type":"ContainerStarted","Data":"8f43193ab7c19d7239bad174ec4d618994850938e1e8dd605e3e6f04e3c44264"} Jul 24 13:39:35.240355 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:35.240281 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" event={"ID":"2254edb8-c574-45db-99a8-320568e891cc","Type":"ContainerStarted","Data":"f7183e8c81cecebda5119c297579f0032e269fcc875476f87ac9aa8d622d1435"} Jul 24 13:39:36.247504 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.247462 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnqvq" event={"ID":"8b70c494-5fd3-4783-a114-907bf3f3805e","Type":"ContainerStarted","Data":"bba01bc6c2ef3b94b1555da4d0cd1159f72848d3bf4269b586039a439d151b72"} Jul 24 13:39:36.249942 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.249913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" event={"ID":"883ffe42-65bc-4a6f-8f14-d4285bc94b90","Type":"ContainerStarted","Data":"823e938f3299f36b8ff2c59660724ff372e4da48e454b705ad45ae9ac1b43e20"} Jul 24 13:39:36.251879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.251845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" event={"ID":"287e3ad6-10ce-4881-96cd-a179071bacb9","Type":"ContainerStarted","Data":"8f161c15e5d80f92058c212fc572a88edccbf8e436474bf6cc9eb03a11e64e47"} Jul 24 13:39:36.252101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.252079 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:36.254371 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.254345 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k8lk6" event={"ID":"d80464ab-57c8-4be8-af66-31519ad6beaa","Type":"ContainerStarted","Data":"99d3d19413701ea8c858e9b39f0f7a675cea88628a84ba42dfe035d84d0926f6"} Jul 24 13:39:36.254481 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.254394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k8lk6" event={"ID":"d80464ab-57c8-4be8-af66-31519ad6beaa","Type":"ContainerStarted","Data":"eafddcea27d88933e779d3123dc679b7a4e57b327ebcefa372a865ae32a8e46d"} Jul 24 13:39:36.254547 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.254494 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:36.265720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.265673 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vnqvq" podStartSLOduration=67.036966634 podStartE2EDuration="1m12.265653123s" totalImagesPullingTime="5.228686489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:36.264627034 +0000 UTC m=+72.875106780" watchObservedRunningTime="2026-07-24 13:39:36.265653123 +0000 UTC m=+72.876132868" Jul 24 13:39:36.284990 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.284937 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" podStartSLOduration=5.284920335 podStartE2EDuration="5.284920335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:36.283673458 +0000 UTC m=+72.894153205" watchObservedRunningTime="2026-07-24 13:39:36.284920335 +0000 UTC m=+72.895400116" Jul 24 13:39:36.306008 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.305953 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bcjmn" podStartSLOduration=36.754687879 podStartE2EDuration="41.305934236s" totalImagesPullingTime="4.551246357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:36.304959699 +0000 UTC m=+72.915439446" watchObservedRunningTime="2026-07-24 13:39:36.305934236 +0000 UTC m=+72.916413983" Jul 24 13:39:36.326462 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:36.326412 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k8lk6" podStartSLOduration=33.895939628 podStartE2EDuration="38.326391298s" totalImagesPullingTime="4.43045167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:36.324899077 +0000 UTC m=+72.935378843" watchObservedRunningTime="2026-07-24 13:39:36.326391298 +0000 UTC m=+72.936871044" Jul 24 13:39:38.264379 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.264290 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-fhj28" event={"ID":"500cd154-214f-47e3-b02e-e5bdeeef3f45","Type":"ContainerStarted","Data":"0503ab2006aa6f8541d86d96ee3dd9edf6b2f69fa9b6f4c227ba82fb6f832f96"} Jul 24 13:39:38.266359 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.266297 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" event={"ID":"49a30465-6733-4640-b754-da43d4c2c0b2","Type":"ContainerStarted","Data":"b34a899c8ad3c56376ab6b5b6bee79981e416230b2c7ca8f8ab7cd3f5ff03c00"} Jul 24 13:39:38.269083 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.269045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" event={"ID":"22bf929a-f86e-40a9-bf36-2a974ed98596","Type":"ContainerStarted","Data":"5b676eeb5c0da00b8aa08e5937c3cbd19b4cb8c7bb2435fc9cfa8cf776cc0b27"} Jul 24 13:39:38.270777 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.270709 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" event={"ID":"2254edb8-c574-45db-99a8-320568e891cc","Type":"ContainerStarted","Data":"62d83ca72469dab2ab8dd0adf6a5b405997d3860221524bd84001bcd1dfad407"} Jul 24 13:39:38.270990 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.270952 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:38.272061 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.272020 2568 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-cqct5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Jul 24 13:39:38.272171 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.272081 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" podUID="2254edb8-c574-45db-99a8-320568e891cc" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Jul 24 13:39:38.288223 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.288177 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-fhj28" podStartSLOduration=2.065620087 podStartE2EDuration="7.288161703s" totalImagesPullingTime="5.222541616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:38.28734769 +0000 UTC m=+74.897827436" watchObservedRunningTime="2026-07-24 13:39:38.288161703 +0000 UTC m=+74.898641481" Jul 24 13:39:38.301758 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.301659 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-25qsv" podStartSLOduration=4.297216787 podStartE2EDuration="7.301636259s" totalImagesPullingTime="3.004419472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:38.301307239 +0000 UTC m=+74.911786988" watchObservedRunningTime="2026-07-24 13:39:38.301636259 +0000 UTC m=+74.912116009" Jul 24 13:39:38.335984 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.335922 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" podStartSLOduration=4.219699091 podStartE2EDuration="7.335898425s" totalImagesPullingTime="3.116199334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:38.33437559 +0000 UTC m=+74.944855338" watchObservedRunningTime="2026-07-24 13:39:38.335898425 +0000 UTC m=+74.946378173" Jul 24 13:39:38.658640 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.658574 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kqlvx" podStartSLOduration=2.439097926 podStartE2EDuration="7.658551476s" totalImagesPullingTime="5.21945355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:38.37613118 +0000 UTC m=+74.986610927" watchObservedRunningTime="2026-07-24 13:39:38.658551476 +0000 UTC m=+75.269031223" Jul 24 13:39:38.659518 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.659497 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq"] Jul 24 13:39:38.661457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.661441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:38.663953 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.663923 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 13:39:38.664082 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.663990 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kjwpf\"" Jul 24 13:39:38.672048 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.672015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq"] Jul 24 13:39:38.752923 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.752883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d43f1ef-1489-44a6-b5f8-ea4ae26dc687-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-542dq\" (UID: \"0d43f1ef-1489-44a6-b5f8-ea4ae26dc687\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:38.853975 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:38.853924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d43f1ef-1489-44a6-b5f8-ea4ae26dc687-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-542dq\" (UID: \"0d43f1ef-1489-44a6-b5f8-ea4ae26dc687\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:38.854150 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:38.854075 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 24 13:39:38.854150 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:38.854140 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d43f1ef-1489-44a6-b5f8-ea4ae26dc687-tls-certificates podName:0d43f1ef-1489-44a6-b5f8-ea4ae26dc687 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:39.354121875 +0000 UTC m=+75.964601624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0d43f1ef-1489-44a6-b5f8-ea4ae26dc687-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-542dq" (UID: "0d43f1ef-1489-44a6-b5f8-ea4ae26dc687") : secret "prometheus-operator-admission-webhook-tls" not found Jul 24 13:39:39.124450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.124403 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r54hx"] Jul 24 13:39:39.127336 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.127269 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.130079 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.130046 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bbx6j\"" Jul 24 13:39:39.130218 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.130106 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 13:39:39.130452 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.130437 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 13:39:39.141653 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.141623 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r54hx"] Jul 24 13:39:39.173785 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.173748 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-f7znd"] Jul 24 13:39:39.175593 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.175570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:39:39.178301 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.178268 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 13:39:39.178545 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.178527 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 13:39:39.178628 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.178546 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vx96w\"" Jul 24 13:39:39.190848 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.190820 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-f7znd"] Jul 24 13:39:39.257001 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.256964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10c4847c-e22b-4313-b08d-e0f477a8c33e-crio-socket\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.257170 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.257012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.257170 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.257041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4lkt\" (UniqueName: \"kubernetes.io/projected/6f5612fb-a3c0-4449-8f18-2b4db6d6252d-kube-api-access-d4lkt\") pod \"downloads-7c97dc4555-f7znd\" (UID: \"6f5612fb-a3c0-4449-8f18-2b4db6d6252d\") " pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:39:39.257170 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.257096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5zv\" (UniqueName: \"kubernetes.io/projected/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-api-access-qz5zv\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.257170 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.257129 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10c4847c-e22b-4313-b08d-e0f477a8c33e-data-volume\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.257170 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.257170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.277896 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.277867 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-cqct5" Jul 24 13:39:39.358337 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10c4847c-e22b-4313-b08d-e0f477a8c33e-crio-socket\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.358607 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358410 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.358607 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358441 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/10c4847c-e22b-4313-b08d-e0f477a8c33e-crio-socket\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.358607 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358460 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4lkt\" (UniqueName: \"kubernetes.io/projected/6f5612fb-a3c0-4449-8f18-2b4db6d6252d-kube-api-access-d4lkt\") pod \"downloads-7c97dc4555-f7znd\" (UID: \"6f5612fb-a3c0-4449-8f18-2b4db6d6252d\") " pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:39:39.358607 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358529 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5zv\" (UniqueName: \"kubernetes.io/projected/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-api-access-qz5zv\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.358607 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10c4847c-e22b-4313-b08d-e0f477a8c33e-data-volume\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.358926 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d43f1ef-1489-44a6-b5f8-ea4ae26dc687-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-542dq\" (UID: \"0d43f1ef-1489-44a6-b5f8-ea4ae26dc687\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:39.358926 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.358677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.358926 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.358774 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:39.358926 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.358850 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:39.858828511 +0000 UTC m=+76.469308254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:39.359098 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.358970 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.359098 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.359019 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:39.859002951 +0000 UTC m=+76.469482699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.359199 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.359182 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/10c4847c-e22b-4313-b08d-e0f477a8c33e-data-volume\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.361153 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.361135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0d43f1ef-1489-44a6-b5f8-ea4ae26dc687-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-542dq\" (UID: \"0d43f1ef-1489-44a6-b5f8-ea4ae26dc687\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:39.367425 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.367399 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4lkt\" (UniqueName: \"kubernetes.io/projected/6f5612fb-a3c0-4449-8f18-2b4db6d6252d-kube-api-access-d4lkt\") pod \"downloads-7c97dc4555-f7znd\" (UID: \"6f5612fb-a3c0-4449-8f18-2b4db6d6252d\") " pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:39:39.367874 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.367854 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5zv\" (UniqueName: \"kubernetes.io/projected/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-api-access-qz5zv\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.489288 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.489195 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:39:39.574061 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.574033 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:39.623479 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.623323 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-f7znd"] Jul 24 13:39:39.628834 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:39.628795 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5612fb_a3c0_4449_8f18_2b4db6d6252d.slice/crio-e6eff2bb12704ddfa7d552b3fa462165e8313e8df478ecf91f72bff624cc981b WatchSource:0}: Error finding container e6eff2bb12704ddfa7d552b3fa462165e8313e8df478ecf91f72bff624cc981b: Status 404 returned error can't find the container with id e6eff2bb12704ddfa7d552b3fa462165e8313e8df478ecf91f72bff624cc981b Jul 24 13:39:39.707560 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.707526 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq"] Jul 24 13:39:39.710462 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:39.710435 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d43f1ef_1489_44a6_b5f8_ea4ae26dc687.slice/crio-648336b5319d8b77cc53aab8ebf5afd42863f0c600d7375bbb8b6a326d7f7472 WatchSource:0}: Error finding container 648336b5319d8b77cc53aab8ebf5afd42863f0c600d7375bbb8b6a326d7f7472: Status 404 returned error can't find the container with id 648336b5319d8b77cc53aab8ebf5afd42863f0c600d7375bbb8b6a326d7f7472 Jul 24 13:39:39.862914 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.862825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.863073 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:39.862909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:39.863073 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.862966 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.863073 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.863015 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:39.863073 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.863030 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:40.863013331 +0000 UTC m=+77.473493056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:39.863073 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:39.863062 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:40.863048506 +0000 UTC m=+77.473528233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:40.134117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:40.134088 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k8lk6_d80464ab-57c8-4be8-af66-31519ad6beaa/dns/0.log" Jul 24 13:39:40.279178 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:40.279134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-f7znd" event={"ID":"6f5612fb-a3c0-4449-8f18-2b4db6d6252d","Type":"ContainerStarted","Data":"e6eff2bb12704ddfa7d552b3fa462165e8313e8df478ecf91f72bff624cc981b"} Jul 24 13:39:40.280415 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:40.280381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" event={"ID":"0d43f1ef-1489-44a6-b5f8-ea4ae26dc687","Type":"ContainerStarted","Data":"648336b5319d8b77cc53aab8ebf5afd42863f0c600d7375bbb8b6a326d7f7472"} Jul 24 13:39:40.326966 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:40.326928 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k8lk6_d80464ab-57c8-4be8-af66-31519ad6beaa/kube-rbac-proxy/0.log" Jul 24 13:39:40.871238 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:40.871177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:40.871430 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:40.871272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:40.871430 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:40.871367 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:40.871430 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:40.871412 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:40.871601 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:40.871422 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:42.871408529 +0000 UTC m=+79.481888253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:40.871601 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:40.871494 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:42.87147479 +0000 UTC m=+79.481954521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:41.116602 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.116571 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c5q46_e83441c2-b725-4664-91d0-5e915d2ba6c1/dns-node-resolver/0.log" Jul 24 13:39:41.287416 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.287305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" event={"ID":"0d43f1ef-1489-44a6-b5f8-ea4ae26dc687","Type":"ContainerStarted","Data":"bf0273e69be5add4c46c6ae135867f477137c5020b80e89379be1050d3870ad0"} Jul 24 13:39:41.287890 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.287511 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:41.292900 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.292871 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" Jul 24 13:39:41.307757 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.307551 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-542dq" podStartSLOduration=2.256361522 podStartE2EDuration="3.307526733s" totalImagesPullingTime="1.051165211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:41.306180177 +0000 UTC m=+77.916659924" watchObservedRunningTime="2026-07-24 13:39:41.307526733 +0000 UTC m=+77.918006480" Jul 24 13:39:41.518283 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.518198 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fc58d8d96-zhw59_287e3ad6-10ce-4881-96cd-a179071bacb9/registry/0.log" Jul 24 13:39:41.918103 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:41.918073 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b984f8d55-8xmml_30ccbce5-4a45-4a1c-8b3f-ce069641e8e3/registry/0.log" Jul 24 13:39:42.317063 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:42.316967 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcsmh_241fa270-01bf-44d5-a857-22457556bd4f/node-ca/0.log" Jul 24 13:39:42.892547 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:42.892494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:42.892777 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:42.892592 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 13:39:42.892777 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:42.892594 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:42.892777 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:42.892665 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:46.892645291 +0000 UTC m=+83.503125015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : configmap "kube-rbac-proxy" not found Jul 24 13:39:42.892777 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:42.892700 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 13:39:42.892957 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:42.892796 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls podName:10c4847c-e22b-4313-b08d-e0f477a8c33e nodeName:}" failed. No retries permitted until 2026-07-24 13:39:46.892778959 +0000 UTC m=+83.503258690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r54hx" (UID: "10c4847c-e22b-4313-b08d-e0f477a8c33e") : secret "insights-runtime-extractor-tls" not found Jul 24 13:39:42.914788 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:42.914758 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4g25d_a40269b1-49be-404b-ab9e-c645775f0848/serve-healthcheck-canary/0.log" Jul 24 13:39:45.089448 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.089408 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-666fz"] Jul 24 13:39:45.093081 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.093055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.095904 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.095879 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 13:39:45.096064 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.095923 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 13:39:45.096064 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.095936 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 13:39:45.096064 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.095962 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 13:39:45.097018 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.097000 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbctl\"" Jul 24 13:39:45.216620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216589 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-sys\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.216620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-wtmp\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.216910 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-textfile\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.216910 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-tls\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.216910 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.216910 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216879 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79570920-ef63-4977-a3ab-01c8047f01a4-metrics-client-ca\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.217117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216929 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.217117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-root\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.217117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.216996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtxt\" (UniqueName: \"kubernetes.io/projected/79570920-ef63-4977-a3ab-01c8047f01a4-kube-api-access-kxtxt\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317519 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79570920-ef63-4977-a3ab-01c8047f01a4-metrics-client-ca\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-root\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtxt\" (UniqueName: \"kubernetes.io/projected/79570920-ef63-4977-a3ab-01c8047f01a4-kube-api-access-kxtxt\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-sys\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-wtmp\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.317712 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-textfile\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318047 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-tls\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318047 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318047 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317907 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-sys\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318047 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.317963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-root\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318251 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.318210 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79570920-ef63-4977-a3ab-01c8047f01a4-metrics-client-ca\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318351 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:45.318324 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 13:39:45.318486 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.318359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-wtmp\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318486 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:45.318399 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-tls podName:79570920-ef63-4977-a3ab-01c8047f01a4 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:45.818380238 +0000 UTC m=+82.428859976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-tls") pod "node-exporter-666fz" (UID: "79570920-ef63-4977-a3ab-01c8047f01a4") : secret "node-exporter-tls" not found Jul 24 13:39:45.318616 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.318554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-textfile\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.318917 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.318894 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.320757 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.320713 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.328836 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.328808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtxt\" (UniqueName: \"kubernetes.io/projected/79570920-ef63-4977-a3ab-01c8047f01a4-kube-api-access-kxtxt\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.823540 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.823160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-tls\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:45.826074 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:45.826015 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/79570920-ef63-4977-a3ab-01c8047f01a4-node-exporter-tls\") pod \"node-exporter-666fz\" (UID: \"79570920-ef63-4977-a3ab-01c8047f01a4\") " pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:46.002928 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.002880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-666fz" Jul 24 13:39:46.157095 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.157056 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:39:46.162648 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.162622 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.165308 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.165283 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 13:39:46.165622 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.165594 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-twmzz\"" Jul 24 13:39:46.165749 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.165625 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 13:39:46.165819 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.165740 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 13:39:46.165819 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.165598 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 13:39:46.166021 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.166003 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 13:39:46.166082 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.166035 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 13:39:46.166133 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.166084 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 13:39:46.166278 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.166247 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 13:39:46.166331 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.166292 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 13:39:46.184582 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.184546 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:39:46.260615 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.260587 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k8lk6" Jul 24 13:39:46.312764 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.312700 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666fz" event={"ID":"79570920-ef63-4977-a3ab-01c8047f01a4","Type":"ContainerStarted","Data":"4c20ad01d46c387814d088942961c5cf35004de1931c108e4499074d3c345d5c"} Jul 24 13:39:46.328701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.328670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.328887 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.328738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.328887 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.328798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.328995 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.328901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.328995 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.328945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329076 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329076 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-config-out\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329164 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329220 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329155 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-config-volume\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329220 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329314 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-web-config\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329314 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pm9\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-kube-api-access-85pm9\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.329403 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.329314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.430772 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.430666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.430772 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.430717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.430792 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.430915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-config-out\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.430941 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.430975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-config-volume\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-web-config\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pm9\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-kube-api-access-85pm9\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431087 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431814 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431814 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431814 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.431814 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.432009 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.431808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.432661 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.432510 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.435944 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.435899 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-config-volume\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.436291 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.436106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.436291 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.436255 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-config-out\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.436890 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.436500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.436890 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.436593 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.436890 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.436850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.437436 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.437391 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.438473 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.438450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-web-config\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.439155 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.439132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.441399 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.441375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pm9\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-kube-api-access-85pm9\") pod \"alertmanager-main-0\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.476182 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.476147 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:39:46.666752 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.666687 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:39:46.938085 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.938036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:46.938286 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.938143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:46.939242 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.939177 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/10c4847c-e22b-4313-b08d-e0f477a8c33e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:46.941367 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:46.941337 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/10c4847c-e22b-4313-b08d-e0f477a8c33e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r54hx\" (UID: \"10c4847c-e22b-4313-b08d-e0f477a8c33e\") " pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:47.239063 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:47.238854 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r54hx" Jul 24 13:39:47.321308 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:47.321184 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"f6b5d0a2372984279eaca9a535fc655fb663ff49f8fba0c342bef60035b11012"} Jul 24 13:39:47.323555 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:47.323526 2568 generic.go:356] "Generic (PLEG): container finished" podID="79570920-ef63-4977-a3ab-01c8047f01a4" containerID="1652bf81d63f705a53af55cf9727f29a3dad35461539888960f6b8b7647d7cf5" exitCode=0 Jul 24 13:39:47.324936 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:47.324605 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666fz" event={"ID":"79570920-ef63-4977-a3ab-01c8047f01a4","Type":"ContainerDied","Data":"1652bf81d63f705a53af55cf9727f29a3dad35461539888960f6b8b7647d7cf5"} Jul 24 13:39:47.416670 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:47.416638 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r54hx"] Jul 24 13:39:47.420412 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:47.420362 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c4847c_e22b_4313_b08d_e0f477a8c33e.slice/crio-f66ae1706de2ce38bb4f5ea6980fbe18feb930bc123f823d90641dba73eccf83 WatchSource:0}: Error finding container f66ae1706de2ce38bb4f5ea6980fbe18feb930bc123f823d90641dba73eccf83: Status 404 returned error can't find the container with id f66ae1706de2ce38bb4f5ea6980fbe18feb930bc123f823d90641dba73eccf83 Jul 24 13:39:48.332762 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:48.332649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r54hx" event={"ID":"10c4847c-e22b-4313-b08d-e0f477a8c33e","Type":"ContainerStarted","Data":"f66ae1706de2ce38bb4f5ea6980fbe18feb930bc123f823d90641dba73eccf83"} Jul 24 13:39:48.334937 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:48.334905 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706" exitCode=0 Jul 24 13:39:48.335818 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:48.335225 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706"} Jul 24 13:39:48.339103 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:48.339035 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666fz" event={"ID":"79570920-ef63-4977-a3ab-01c8047f01a4","Type":"ContainerStarted","Data":"a835e93a67c7cea58ec463853f5f093f5779ac4d06fdc902b96016e4acd78713"} Jul 24 13:39:48.339103 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:48.339085 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-666fz" event={"ID":"79570920-ef63-4977-a3ab-01c8047f01a4","Type":"ContainerStarted","Data":"154a9fd3fd75656869a72972e7504e6a386dd709159196f74647f4c3d20179b8"} Jul 24 13:39:48.393061 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:48.392996 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-666fz" podStartSLOduration=2.214768899 podStartE2EDuration="3.39297495s" totalImagesPullingTime="1.09659559s" totalInitContainerRuntime="81.610461ms" isStatefulPod=true podCreationTimestamp="2026-07-24 13:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:39:48.391276723 +0000 UTC m=+85.001756470" watchObservedRunningTime="2026-07-24 13:39:48.39297495 +0000 UTC m=+85.003454696" Jul 24 13:39:49.505452 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.505413 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67495bb879-g42z7"] Jul 24 13:39:49.510201 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.509384 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.513100 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.512785 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 13:39:49.514391 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.513350 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 13:39:49.514391 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.513956 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 13:39:49.514391 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.514244 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fdw6c\"" Jul 24 13:39:49.515178 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.514852 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3an86n7fkm9jv\"" Jul 24 13:39:49.515178 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.515057 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 13:39:49.522035 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.521446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67495bb879-g42z7"] Jul 24 13:39:49.664411 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-secret-metrics-server-client-certs\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.664577 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb93816-b770-4025-94da-7bc8bbe942dd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.664577 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-secret-metrics-server-tls\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.664577 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-client-ca-bundle\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.664678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664592 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abb93816-b770-4025-94da-7bc8bbe942dd-audit-log\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.664678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abb93816-b770-4025-94da-7bc8bbe942dd-metrics-server-audit-profiles\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.664678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.664652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcv2\" (UniqueName: \"kubernetes.io/projected/abb93816-b770-4025-94da-7bc8bbe942dd-kube-api-access-6dcv2\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.766224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb93816-b770-4025-94da-7bc8bbe942dd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.766224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-secret-metrics-server-tls\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.766224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-client-ca-bundle\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.766224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abb93816-b770-4025-94da-7bc8bbe942dd-audit-log\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.767209 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abb93816-b770-4025-94da-7bc8bbe942dd-metrics-server-audit-profiles\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.767209 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcv2\" (UniqueName: \"kubernetes.io/projected/abb93816-b770-4025-94da-7bc8bbe942dd-kube-api-access-6dcv2\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.767209 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.766868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-secret-metrics-server-client-certs\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.767479 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.767263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abb93816-b770-4025-94da-7bc8bbe942dd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.768162 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.768134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abb93816-b770-4025-94da-7bc8bbe942dd-metrics-server-audit-profiles\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.768623 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.768574 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abb93816-b770-4025-94da-7bc8bbe942dd-audit-log\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.770271 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.770248 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-client-ca-bundle\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.771414 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.771390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-secret-metrics-server-client-certs\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.772023 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.771982 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abb93816-b770-4025-94da-7bc8bbe942dd-secret-metrics-server-tls\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.781436 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.781382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcv2\" (UniqueName: \"kubernetes.io/projected/abb93816-b770-4025-94da-7bc8bbe942dd-kube-api-access-6dcv2\") pod \"metrics-server-67495bb879-g42z7\" (UID: \"abb93816-b770-4025-94da-7bc8bbe942dd\") " pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.826402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.826122 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:39:49.897315 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.895805 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9"] Jul 24 13:39:49.899595 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.899568 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:49.902523 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.902482 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 13:39:49.902719 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.902701 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ng7gm\"" Jul 24 13:39:49.909015 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.908981 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9"] Jul 24 13:39:49.918055 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.918028 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55ddfc5568-6kkn8"] Jul 24 13:39:49.922219 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.922194 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:49.925697 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.925661 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 13:39:49.926916 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.926366 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 13:39:49.926916 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.926371 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hqg9d\"" Jul 24 13:39:49.926916 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.926710 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 13:39:49.926916 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.926773 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 13:39:49.927154 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.926997 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 13:39:49.935699 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:49.935670 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ddfc5568-6kkn8"] Jul 24 13:39:50.069478 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-oauth-config\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.069645 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069505 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-serving-cert\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.069645 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069569 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-config\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.069859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-oauth-serving-cert\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.069859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-service-ca\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.069859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d688c9f8-373a-465a-98c3-76f1d9488f74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.069859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfrkp\" (UniqueName: \"kubernetes.io/projected/d688c9f8-373a-465a-98c3-76f1d9488f74-kube-api-access-dfrkp\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.069859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.069861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jb5\" (UniqueName: \"kubernetes.io/projected/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-kube-api-access-28jb5\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.145751 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.145660 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67495bb879-g42z7"] Jul 24 13:39:50.149509 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:50.149473 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb93816_b770_4025_94da_7bc8bbe942dd.slice/crio-69f07b0889c9462748be00d813a1b1afe53cee2fed72f5dab2335a3c25e09818 WatchSource:0}: Error finding container 69f07b0889c9462748be00d813a1b1afe53cee2fed72f5dab2335a3c25e09818: Status 404 returned error can't find the container with id 69f07b0889c9462748be00d813a1b1afe53cee2fed72f5dab2335a3c25e09818 Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jb5\" (UniqueName: \"kubernetes.io/projected/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-kube-api-access-28jb5\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171155 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-oauth-config\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-serving-cert\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-config\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-oauth-serving-cert\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171319 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-service-ca\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d688c9f8-373a-465a-98c3-76f1d9488f74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.171400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrkp\" (UniqueName: \"kubernetes.io/projected/d688c9f8-373a-465a-98c3-76f1d9488f74-kube-api-access-dfrkp\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:50.172013 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 24 13:39:50.172307 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:39:50.172095 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d688c9f8-373a-465a-98c3-76f1d9488f74-monitoring-plugin-cert podName:d688c9f8-373a-465a-98c3-76f1d9488f74 nodeName:}" failed. No retries permitted until 2026-07-24 13:39:50.672072422 +0000 UTC m=+87.282552160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d688c9f8-373a-465a-98c3-76f1d9488f74-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-lx9g9" (UID: "d688c9f8-373a-465a-98c3-76f1d9488f74") : secret "monitoring-plugin-cert" not found Jul 24 13:39:50.172915 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.172517 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-service-ca\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172915 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.172815 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-config\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.172915 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.172896 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-oauth-serving-cert\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.174907 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.174864 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-oauth-config\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.175932 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.175882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-serving-cert\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.184914 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.184886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrkp\" (UniqueName: \"kubernetes.io/projected/d688c9f8-373a-465a-98c3-76f1d9488f74-kube-api-access-dfrkp\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.186660 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.186456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jb5\" (UniqueName: \"kubernetes.io/projected/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-kube-api-access-28jb5\") pod \"console-55ddfc5568-6kkn8\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.238609 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.238418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:39:50.362650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.362455 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836"} Jul 24 13:39:50.362650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.362541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328"} Jul 24 13:39:50.362650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.362582 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62"} Jul 24 13:39:50.364303 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.364234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" event={"ID":"abb93816-b770-4025-94da-7bc8bbe942dd","Type":"ContainerStarted","Data":"69f07b0889c9462748be00d813a1b1afe53cee2fed72f5dab2335a3c25e09818"} Jul 24 13:39:50.410413 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.410379 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55ddfc5568-6kkn8"] Jul 24 13:39:50.676691 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.676649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d688c9f8-373a-465a-98c3-76f1d9488f74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.679485 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.679456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d688c9f8-373a-465a-98c3-76f1d9488f74-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-lx9g9\" (UID: \"d688c9f8-373a-465a-98c3-76f1d9488f74\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:50.814960 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:50.814922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:39:51.372640 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:51.372608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608"} Jul 24 13:39:51.895066 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:51.895031 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b969ddcd9-8rnjl"] Jul 24 13:39:51.913126 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:51.913085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b969ddcd9-8rnjl"] Jul 24 13:39:51.913299 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:51.913219 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:51.922212 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:51.922176 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 13:39:52.089014 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.088970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2xf\" (UniqueName: \"kubernetes.io/projected/9db948d2-4b5b-435c-8875-49b2e9979130-kube-api-access-zc2xf\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.089192 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.089029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-trusted-ca-bundle\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.089192 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.089090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-serving-cert\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.089192 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.089111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-service-ca\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.089192 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.089176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-oauth-config\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.089413 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.089225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-oauth-serving-cert\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.089413 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.089258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-console-config\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190371 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-oauth-config\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190371 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-oauth-serving-cert\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-console-config\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2xf\" (UniqueName: \"kubernetes.io/projected/9db948d2-4b5b-435c-8875-49b2e9979130-kube-api-access-zc2xf\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190469 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-trusted-ca-bundle\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-serving-cert\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.190604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.190542 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-service-ca\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.191263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.191197 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-oauth-serving-cert\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.191394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.191345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-service-ca\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.191394 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.191345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-console-config\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.191597 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.191569 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-trusted-ca-bundle\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.193274 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.193251 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-serving-cert\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.193376 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.193281 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-oauth-config\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.200287 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.200242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2xf\" (UniqueName: \"kubernetes.io/projected/9db948d2-4b5b-435c-8875-49b2e9979130-kube-api-access-zc2xf\") pod \"console-5b969ddcd9-8rnjl\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.225825 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:52.225785 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:39:52.468605 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:39:52.468525 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac7e6f8_46c6_400f_b4bf_ab4749aba609.slice/crio-8e85b2f34de8df2551a11a704055e9487629544005b703cd22815c107e10146a WatchSource:0}: Error finding container 8e85b2f34de8df2551a11a704055e9487629544005b703cd22815c107e10146a: Status 404 returned error can't find the container with id 8e85b2f34de8df2551a11a704055e9487629544005b703cd22815c107e10146a Jul 24 13:39:53.219219 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:53.219189 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:39:53.383016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:53.382973 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ddfc5568-6kkn8" event={"ID":"2ac7e6f8-46c6-400f-b4bf-ab4749aba609","Type":"ContainerStarted","Data":"8e85b2f34de8df2551a11a704055e9487629544005b703cd22815c107e10146a"} Jul 24 13:39:57.263846 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:57.263811 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fc58d8d96-zhw59" Jul 24 13:39:58.233886 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:39:58.233834 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" podUID="30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" containerName="registry" containerID="cri-o://22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa" gracePeriod=30 Jul 24 13:40:00.234802 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.234741 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273173 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-trusted-ca\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273231 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxqr5\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-kube-api-access-qxqr5\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273269 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-bound-sa-token\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273321 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273355 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-certificates\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273388 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-installation-pull-secrets\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273426 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-image-registry-private-configuration\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.273471 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-ca-trust-extracted\") pod \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\" (UID: \"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3\") " Jul 24 13:40:00.274701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.274295 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:00.274791 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.274700 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:00.281457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.281401 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-kube-api-access-qxqr5" (OuterVolumeSpecName: "kube-api-access-qxqr5") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "kube-api-access-qxqr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:40:00.281765 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.281711 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:40:00.281855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.281830 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:40:00.281949 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.281926 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:40:00.282406 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.282379 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:40:00.286818 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.286702 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" (UID: "30ccbce5-4a45-4a1c-8b3f-ce069641e8e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:40:00.375128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375086 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-trusted-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375126 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqr5\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-kube-api-access-qxqr5\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375143 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-bound-sa-token\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375159 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-tls\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375172 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-registry-certificates\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375185 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-installation-pull-secrets\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375201 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-image-registry-private-configuration\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.375457 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.375215 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3-ca-trust-extracted\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:00.417635 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.417442 2568 generic.go:356] "Generic (PLEG): container finished" podID="30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" containerID="22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa" exitCode=0 Jul 24 13:40:00.417635 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.417498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" event={"ID":"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3","Type":"ContainerDied","Data":"22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa"} Jul 24 13:40:00.417635 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.417518 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" Jul 24 13:40:00.417635 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.417555 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b984f8d55-8xmml" event={"ID":"30ccbce5-4a45-4a1c-8b3f-ce069641e8e3","Type":"ContainerDied","Data":"425275734bfc016e182b6412f2186081dec34e31b16566ed4cef095ec5081a3a"} Jul 24 13:40:00.418409 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.418254 2568 scope.go:117] "RemoveContainer" containerID="22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa" Jul 24 13:40:00.444493 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.444453 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b984f8d55-8xmml"] Jul 24 13:40:00.451379 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.451305 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b984f8d55-8xmml"] Jul 24 13:40:00.774677 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.774609 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b969ddcd9-8rnjl"] Jul 24 13:40:00.788697 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:00.788653 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9"] Jul 24 13:40:01.121865 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:01.121716 2568 scope.go:117] "RemoveContainer" containerID="22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa" Jul 24 13:40:01.122697 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:40:01.122593 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa\": container with ID starting with 22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa not found: ID does not exist" containerID="22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa" Jul 24 13:40:01.122697 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:01.122647 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa"} err="failed to get container status \"22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa\": rpc error: code = NotFound desc = could not find container \"22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa\": container with ID starting with 22a343ef4a92cbacc4a848a23b7d268c523020f97d0e64993969509d9049a2fa not found: ID does not exist" Jul 24 13:40:01.123100 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:40:01.123067 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db948d2_4b5b_435c_8875_49b2e9979130.slice/crio-d5a3866bdc6020641e01110e74fdc99fb2579de97539e26b963bbfcde0698171 WatchSource:0}: Error finding container d5a3866bdc6020641e01110e74fdc99fb2579de97539e26b963bbfcde0698171: Status 404 returned error can't find the container with id d5a3866bdc6020641e01110e74fdc99fb2579de97539e26b963bbfcde0698171 Jul 24 13:40:01.124325 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:40:01.124299 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd688c9f8_373a_465a_98c3_76f1d9488f74.slice/crio-c878f10e729aa14c135b450faeeee250ccace1b8d47dac1b1058205a86dd3fa8 WatchSource:0}: Error finding container c878f10e729aa14c135b450faeeee250ccace1b8d47dac1b1058205a86dd3fa8: Status 404 returned error can't find the container with id c878f10e729aa14c135b450faeeee250ccace1b8d47dac1b1058205a86dd3fa8 Jul 24 13:40:01.424082 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:01.423987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b969ddcd9-8rnjl" event={"ID":"9db948d2-4b5b-435c-8875-49b2e9979130","Type":"ContainerStarted","Data":"d5a3866bdc6020641e01110e74fdc99fb2579de97539e26b963bbfcde0698171"} Jul 24 13:40:01.425362 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:01.425327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" event={"ID":"d688c9f8-373a-465a-98c3-76f1d9488f74","Type":"ContainerStarted","Data":"c878f10e729aa14c135b450faeeee250ccace1b8d47dac1b1058205a86dd3fa8"} Jul 24 13:40:01.429136 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:01.429104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681"} Jul 24 13:40:01.860718 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:01.860634 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" path="/var/lib/kubelet/pods/30ccbce5-4a45-4a1c-8b3f-ce069641e8e3/volumes" Jul 24 13:40:02.208198 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:02.208139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7j8ww" Jul 24 13:40:03.464709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.464654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" event={"ID":"abb93816-b770-4025-94da-7bc8bbe942dd","Type":"ContainerStarted","Data":"25e694e8242ebfd0f91ec52268c27c61e28899b9ac3841cadf540adb40f56810"} Jul 24 13:40:03.469045 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.468975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r54hx" event={"ID":"10c4847c-e22b-4313-b08d-e0f477a8c33e","Type":"ContainerStarted","Data":"9e2877644e272f388cd2298ec397c689b48e6fc85dae0e50af4df281c5d42fca"} Jul 24 13:40:03.473165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.473132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b969ddcd9-8rnjl" event={"ID":"9db948d2-4b5b-435c-8875-49b2e9979130","Type":"ContainerStarted","Data":"a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646"} Jul 24 13:40:03.478021 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.477989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ddfc5568-6kkn8" event={"ID":"2ac7e6f8-46c6-400f-b4bf-ab4749aba609","Type":"ContainerStarted","Data":"bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b"} Jul 24 13:40:03.481199 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.481134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-f7znd" event={"ID":"6f5612fb-a3c0-4449-8f18-2b4db6d6252d","Type":"ContainerStarted","Data":"a9606ed30af3ea0a9ab567ff07e27cc169ef7b898cd963c859ecd43bbb64022a"} Jul 24 13:40:03.481456 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.481440 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:40:03.499909 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.499819 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-f7znd" Jul 24 13:40:03.502715 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.502658 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" podStartSLOduration=3.48655495 podStartE2EDuration="14.502636253s" totalImagesPullingTime="11.016081303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:03.485352881 +0000 UTC m=+100.095832655" watchObservedRunningTime="2026-07-24 13:40:03.502636253 +0000 UTC m=+100.113116001" Jul 24 13:40:03.504043 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.503996 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-f7znd" podStartSLOduration=2.9858554699999997 podStartE2EDuration="24.5039787s" totalImagesPullingTime="21.51812323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:03.501920657 +0000 UTC m=+100.112400419" watchObservedRunningTime="2026-07-24 13:40:03.5039787 +0000 UTC m=+100.114458447" Jul 24 13:40:03.521102 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.521043 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b969ddcd9-8rnjl" podStartSLOduration=10.94958803 podStartE2EDuration="12.520998895s" totalImagesPullingTime="1.571410865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:03.519639974 +0000 UTC m=+100.130119721" watchObservedRunningTime="2026-07-24 13:40:03.520998895 +0000 UTC m=+100.131478641" Jul 24 13:40:03.541088 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:03.540615 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55ddfc5568-6kkn8" podStartSLOduration=4.278147074 podStartE2EDuration="14.540590576s" totalImagesPullingTime="10.262443502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:03.538357207 +0000 UTC m=+100.148836958" watchObservedRunningTime="2026-07-24 13:40:03.540590576 +0000 UTC m=+100.151070326" Jul 24 13:40:05.500704 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:05.500666 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerStarted","Data":"54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328"} Jul 24 13:40:05.529554 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:05.529501 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=13.126366584 podStartE2EDuration="19.5294802s" totalImagesPullingTime="6.384071211s" totalInitContainerRuntime="19.042405ms" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:05.527447388 +0000 UTC m=+102.137927158" watchObservedRunningTime="2026-07-24 13:40:05.5294802 +0000 UTC m=+102.139959947" Jul 24 13:40:08.533398 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:08.533362 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r54hx" event={"ID":"10c4847c-e22b-4313-b08d-e0f477a8c33e","Type":"ContainerStarted","Data":"4ea0aa55ec09aec36179ba4c37396ae963e3fc151dbd69f4ac05b7c7d774266e"} Jul 24 13:40:08.535243 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:08.535215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" event={"ID":"d688c9f8-373a-465a-98c3-76f1d9488f74","Type":"ContainerStarted","Data":"dd49c44232e62f44e44d7f4fa9649d39ba0fc7bbb6225068c1d55190181a0128"} Jul 24 13:40:08.535430 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:08.535408 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:40:08.541443 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:08.541417 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" Jul 24 13:40:08.552740 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:08.552670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-lx9g9" podStartSLOduration=12.764020547 podStartE2EDuration="19.552654658s" totalImagesPullingTime="6.788634111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:08.550477479 +0000 UTC m=+105.160957226" watchObservedRunningTime="2026-07-24 13:40:08.552654658 +0000 UTC m=+105.163134404" Jul 24 13:40:09.826898 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:09.826804 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:40:09.826898 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:09.826884 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:40:10.239438 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:10.239397 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:40:10.240437 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:10.240412 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:40:10.247124 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:10.247084 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:40:10.552031 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:10.551919 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:40:12.226378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:12.226346 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:40:12.226869 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:12.226425 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:40:12.232969 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:12.232934 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:40:12.560516 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:12.560436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:40:12.609388 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:12.609358 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ddfc5568-6kkn8"] Jul 24 13:40:13.562581 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:13.562466 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r54hx" event={"ID":"10c4847c-e22b-4313-b08d-e0f477a8c33e","Type":"ContainerStarted","Data":"e4e19b86cecb40b4d48923d080b154805bde96fe8711de66f38d9adb857b0754"} Jul 24 13:40:13.579801 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:13.579749 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r54hx" podStartSLOduration=10.285546113 podStartE2EDuration="34.579712149s" totalImagesPullingTime="24.294166036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 13:39:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:40:13.578228332 +0000 UTC m=+110.188708081" watchObservedRunningTime="2026-07-24 13:40:13.579712149 +0000 UTC m=+110.190191894" Jul 24 13:40:16.574903 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:16.574866 2568 generic.go:356] "Generic (PLEG): container finished" podID="e0108565-5b1f-4f9a-b7d5-f12319bf0358" containerID="5bae1e21e7c3cd27e2a606815b177c93ab67c60d91fa18f9752daa12ee94bf42" exitCode=0 Jul 24 13:40:16.575286 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:16.574938 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" event={"ID":"e0108565-5b1f-4f9a-b7d5-f12319bf0358","Type":"ContainerDied","Data":"5bae1e21e7c3cd27e2a606815b177c93ab67c60d91fa18f9752daa12ee94bf42"} Jul 24 13:40:16.575286 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:16.575266 2568 scope.go:117] "RemoveContainer" containerID="5bae1e21e7c3cd27e2a606815b177c93ab67c60d91fa18f9752daa12ee94bf42" Jul 24 13:40:17.581789 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:17.581753 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qxjld" event={"ID":"e0108565-5b1f-4f9a-b7d5-f12319bf0358","Type":"ContainerStarted","Data":"68d0a2a015423f38d6f8119a245fa4f2a25b5c4b60d5497e18bff7e6e5c5ea1e"} Jul 24 13:40:26.618320 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:26.618281 2568 generic.go:356] "Generic (PLEG): container finished" podID="afe86dfe-2fb9-46b8-a96c-876b4f8b1507" containerID="0db3c2ea0f4099806834127ceb9d72702dc369f886be84ed941e2aa8509c465a" exitCode=0 Jul 24 13:40:26.618754 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:26.618352 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" event={"ID":"afe86dfe-2fb9-46b8-a96c-876b4f8b1507","Type":"ContainerDied","Data":"0db3c2ea0f4099806834127ceb9d72702dc369f886be84ed941e2aa8509c465a"} Jul 24 13:40:26.618754 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:26.618707 2568 scope.go:117] "RemoveContainer" containerID="0db3c2ea0f4099806834127ceb9d72702dc369f886be84ed941e2aa8509c465a" Jul 24 13:40:27.623993 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:27.623954 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wbpls" event={"ID":"afe86dfe-2fb9-46b8-a96c-876b4f8b1507","Type":"ContainerStarted","Data":"21ecf3fcaae5beed8c0671b0d2d143c0ac0e0004ea53c9cac63ee26a5adb8bbe"} Jul 24 13:40:29.832190 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:29.832156 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:40:29.836115 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:29.836083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67495bb879-g42z7" Jul 24 13:40:38.581667 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.581598 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55ddfc5568-6kkn8" podUID="2ac7e6f8-46c6-400f-b4bf-ab4749aba609" containerName="console" containerID="cri-o://bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b" gracePeriod=15 Jul 24 13:40:38.859507 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.859434 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:40:38.940009 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.939966 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-config\") pod \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " Jul 24 13:40:38.940203 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940023 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-serving-cert\") pod \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " Jul 24 13:40:38.940203 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940070 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-service-ca\") pod \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " Jul 24 13:40:38.940203 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940117 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-oauth-serving-cert\") pod \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " Jul 24 13:40:38.940203 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940154 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-oauth-config\") pod \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " Jul 24 13:40:38.940203 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940182 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28jb5\" (UniqueName: \"kubernetes.io/projected/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-kube-api-access-28jb5\") pod \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\" (UID: \"2ac7e6f8-46c6-400f-b4bf-ab4749aba609\") " Jul 24 13:40:38.940501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940470 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-config" (OuterVolumeSpecName: "console-config") pod "2ac7e6f8-46c6-400f-b4bf-ab4749aba609" (UID: "2ac7e6f8-46c6-400f-b4bf-ab4749aba609"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:38.940557 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940478 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ac7e6f8-46c6-400f-b4bf-ab4749aba609" (UID: "2ac7e6f8-46c6-400f-b4bf-ab4749aba609"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:38.940557 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.940480 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ac7e6f8-46c6-400f-b4bf-ab4749aba609" (UID: "2ac7e6f8-46c6-400f-b4bf-ab4749aba609"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:40:38.942450 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.942416 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ac7e6f8-46c6-400f-b4bf-ab4749aba609" (UID: "2ac7e6f8-46c6-400f-b4bf-ab4749aba609"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:40:38.942595 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.942537 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ac7e6f8-46c6-400f-b4bf-ab4749aba609" (UID: "2ac7e6f8-46c6-400f-b4bf-ab4749aba609"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:40:38.942595 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:38.942562 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-kube-api-access-28jb5" (OuterVolumeSpecName: "kube-api-access-28jb5") pod "2ac7e6f8-46c6-400f-b4bf-ab4749aba609" (UID: "2ac7e6f8-46c6-400f-b4bf-ab4749aba609"). InnerVolumeSpecName "kube-api-access-28jb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:40:39.041313 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.041270 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:39.041313 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.041305 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:39.041313 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.041318 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-oauth-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:39.041561 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.041329 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-oauth-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:39.041561 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.041351 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jb5\" (UniqueName: \"kubernetes.io/projected/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-kube-api-access-28jb5\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:39.041561 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.041363 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ac7e6f8-46c6-400f-b4bf-ab4749aba609-console-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:40:39.674520 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.674478 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ac7e6f8-46c6-400f-b4bf-ab4749aba609" containerID="bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b" exitCode=0 Jul 24 13:40:39.674955 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.674546 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55ddfc5568-6kkn8" Jul 24 13:40:39.674955 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.674567 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ddfc5568-6kkn8" event={"ID":"2ac7e6f8-46c6-400f-b4bf-ab4749aba609","Type":"ContainerDied","Data":"bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b"} Jul 24 13:40:39.674955 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.674647 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55ddfc5568-6kkn8" event={"ID":"2ac7e6f8-46c6-400f-b4bf-ab4749aba609","Type":"ContainerDied","Data":"8e85b2f34de8df2551a11a704055e9487629544005b703cd22815c107e10146a"} Jul 24 13:40:39.674955 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.674667 2568 scope.go:117] "RemoveContainer" containerID="bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b" Jul 24 13:40:39.683380 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.683358 2568 scope.go:117] "RemoveContainer" containerID="bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b" Jul 24 13:40:39.683657 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:40:39.683637 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b\": container with ID starting with bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b not found: ID does not exist" containerID="bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b" Jul 24 13:40:39.683748 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.683666 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b"} err="failed to get container status \"bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b\": rpc error: code = NotFound desc = could not find container \"bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b\": container with ID starting with bb3190d0483bcc7d8e93afcea06e43f45094c5d185e844083c3df3a2ef8efc4b not found: ID does not exist" Jul 24 13:40:39.696334 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.696307 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55ddfc5568-6kkn8"] Jul 24 13:40:39.699091 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.699067 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55ddfc5568-6kkn8"] Jul 24 13:40:39.858716 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:40:39.858683 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ac7e6f8-46c6-400f-b4bf-ab4749aba609" path="/var/lib/kubelet/pods/2ac7e6f8-46c6-400f-b4bf-ab4749aba609/volumes" Jul 24 13:41:05.433933 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.433899 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:41:05.434412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.434198 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="alertmanager" containerID="cri-o://1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62" gracePeriod=120 Jul 24 13:41:05.434412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.434279 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-web" containerID="cri-o://271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836" gracePeriod=120 Jul 24 13:41:05.434412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.434268 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-metric" containerID="cri-o://b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681" gracePeriod=120 Jul 24 13:41:05.434412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.434329 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy" containerID="cri-o://016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608" gracePeriod=120 Jul 24 13:41:05.434412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.434343 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="prom-label-proxy" containerID="cri-o://54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328" gracePeriod=120 Jul 24 13:41:05.434412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.434294 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="config-reloader" containerID="cri-o://d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328" gracePeriod=120 Jul 24 13:41:05.782558 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782470 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328" exitCode=0 Jul 24 13:41:05.782558 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782500 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681" exitCode=0 Jul 24 13:41:05.782558 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782508 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608" exitCode=0 Jul 24 13:41:05.782558 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782516 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328" exitCode=0 Jul 24 13:41:05.782558 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782522 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62" exitCode=0 Jul 24 13:41:05.782841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328"} Jul 24 13:41:05.782841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681"} Jul 24 13:41:05.782841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782668 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608"} Jul 24 13:41:05.782841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782691 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328"} Jul 24 13:41:05.782841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:05.782715 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62"} Jul 24 13:41:06.681033 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.681006 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:06.789802 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.789700 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-web-config\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.789802 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.789793 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790037 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.789819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-web\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790037 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.789836 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-cluster-tls-config\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790037 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.789867 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-main-db\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790037 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.789910 2568 generic.go:356] "Generic (PLEG): container finished" podID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerID="271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836" exitCode=0 Jul 24 13:41:06.790235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790033 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:06.790235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790082 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-tls-assets\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790116 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-config-out\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790145 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-config-volume\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790172 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:41:06.790235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790193 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85pm9\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-kube-api-access-85pm9\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790511 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790240 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-trusted-ca-bundle\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790511 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790292 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-main-tls\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790511 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790319 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-metrics-client-ca\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790511 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790346 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"86620919-a5dc-4674-a6cf-4cee43cdccda\" (UID: \"86620919-a5dc-4674-a6cf-4cee43cdccda\") " Jul 24 13:41:06.790685 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790040 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836"} Jul 24 13:41:06.790685 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790651 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-main-db\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.790827 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"86620919-a5dc-4674-a6cf-4cee43cdccda","Type":"ContainerDied","Data":"f6b5d0a2372984279eaca9a535fc655fb663ff49f8fba0c342bef60035b11012"} Jul 24 13:41:06.790827 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.790802 2568 scope.go:117] "RemoveContainer" containerID="54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328" Jul 24 13:41:06.792958 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.792920 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:06.793438 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.793406 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:06.794028 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.793989 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-config-out" (OuterVolumeSpecName: "config-out") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:41:06.794120 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.794020 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:06.794187 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.794152 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.794251 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.794186 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.794806 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.794780 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-kube-api-access-85pm9" (OuterVolumeSpecName: "kube-api-access-85pm9") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "kube-api-access-85pm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:06.794990 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.794942 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.795101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.795008 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-config-volume" (OuterVolumeSpecName: "config-volume") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.795101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.795072 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.798898 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.798719 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.805101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.805076 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-web-config" (OuterVolumeSpecName: "web-config") pod "86620919-a5dc-4674-a6cf-4cee43cdccda" (UID: "86620919-a5dc-4674-a6cf-4cee43cdccda"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:06.814259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.814237 2568 scope.go:117] "RemoveContainer" containerID="b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681" Jul 24 13:41:06.821524 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.821501 2568 scope.go:117] "RemoveContainer" containerID="016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608" Jul 24 13:41:06.828252 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.828235 2568 scope.go:117] "RemoveContainer" containerID="271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836" Jul 24 13:41:06.835259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.835236 2568 scope.go:117] "RemoveContainer" containerID="d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328" Jul 24 13:41:06.843993 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.843971 2568 scope.go:117] "RemoveContainer" containerID="1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62" Jul 24 13:41:06.852139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.852118 2568 scope.go:117] "RemoveContainer" containerID="c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706" Jul 24 13:41:06.859080 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859059 2568 scope.go:117] "RemoveContainer" containerID="54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328" Jul 24 13:41:06.859328 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.859307 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328\": container with ID starting with 54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328 not found: ID does not exist" containerID="54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328" Jul 24 13:41:06.859406 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859339 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328"} err="failed to get container status \"54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328\": rpc error: code = NotFound desc = could not find container \"54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328\": container with ID starting with 54ccc512adb281094c616f09617505ee956e1cd807c28606ff177c2ff2866328 not found: ID does not exist" Jul 24 13:41:06.859406 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859366 2568 scope.go:117] "RemoveContainer" containerID="b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681" Jul 24 13:41:06.859642 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.859625 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681\": container with ID starting with b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681 not found: ID does not exist" containerID="b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681" Jul 24 13:41:06.859678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859649 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681"} err="failed to get container status \"b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681\": rpc error: code = NotFound desc = could not find container \"b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681\": container with ID starting with b665e782f6aac272299351cb7ebe02733a81e369d2524b18ba8bec6517aae681 not found: ID does not exist" Jul 24 13:41:06.859678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859665 2568 scope.go:117] "RemoveContainer" containerID="016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608" Jul 24 13:41:06.859930 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.859913 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608\": container with ID starting with 016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608 not found: ID does not exist" containerID="016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608" Jul 24 13:41:06.859998 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859935 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608"} err="failed to get container status \"016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608\": rpc error: code = NotFound desc = could not find container \"016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608\": container with ID starting with 016a596e67816e9c2192b8c12a5d5b4cedd07f83b7b82a4f0e4326919c92e608 not found: ID does not exist" Jul 24 13:41:06.859998 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.859950 2568 scope.go:117] "RemoveContainer" containerID="271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836" Jul 24 13:41:06.860214 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.860197 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836\": container with ID starting with 271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836 not found: ID does not exist" containerID="271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836" Jul 24 13:41:06.860263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860228 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836"} err="failed to get container status \"271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836\": rpc error: code = NotFound desc = could not find container \"271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836\": container with ID starting with 271e07c0cda044d02a969d2479351fa23bc9bd1a775f070e8d6a818f77960836 not found: ID does not exist" Jul 24 13:41:06.860263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860243 2568 scope.go:117] "RemoveContainer" containerID="d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328" Jul 24 13:41:06.860477 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.860461 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328\": container with ID starting with d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328 not found: ID does not exist" containerID="d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328" Jul 24 13:41:06.860518 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860481 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328"} err="failed to get container status \"d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328\": rpc error: code = NotFound desc = could not find container \"d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328\": container with ID starting with d06b2f338cf2a18052de697a2741408b7901fddeb472b51eb00a888c7dd45328 not found: ID does not exist" Jul 24 13:41:06.860518 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860495 2568 scope.go:117] "RemoveContainer" containerID="1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62" Jul 24 13:41:06.860700 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.860684 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62\": container with ID starting with 1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62 not found: ID does not exist" containerID="1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62" Jul 24 13:41:06.860769 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860705 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62"} err="failed to get container status \"1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62\": rpc error: code = NotFound desc = could not find container \"1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62\": container with ID starting with 1462078961bc48b31b8cf7d9a81c3216a3b4665baadbba936d334b464cd00e62 not found: ID does not exist" Jul 24 13:41:06.860769 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860739 2568 scope.go:117] "RemoveContainer" containerID="c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706" Jul 24 13:41:06.860971 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:06.860952 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706\": container with ID starting with c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706 not found: ID does not exist" containerID="c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706" Jul 24 13:41:06.861011 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.860976 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706"} err="failed to get container status \"c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706\": rpc error: code = NotFound desc = could not find container \"c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706\": container with ID starting with c498ec0735cb36d08ae41afc6a9026a5b810eb2924b428c516f12319603ae706 not found: ID does not exist" Jul 24 13:41:06.891444 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891411 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891444 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891439 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891444 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891451 2568 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-cluster-tls-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891462 2568 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-tls-assets\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891470 2568 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86620919-a5dc-4674-a6cf-4cee43cdccda-config-out\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891479 2568 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891486 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pm9\" (UniqueName: \"kubernetes.io/projected/86620919-a5dc-4674-a6cf-4cee43cdccda-kube-api-access-85pm9\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891495 2568 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891503 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-main-tls\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891512 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86620919-a5dc-4674-a6cf-4cee43cdccda-metrics-client-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891520 2568 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:06.891643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:06.891528 2568 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86620919-a5dc-4674-a6cf-4cee43cdccda-web-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:07.121424 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.121388 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:41:07.125337 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.125310 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:41:07.154911 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.154871 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:41:07.155230 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155215 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-web" Jul 24 13:41:07.155230 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155232 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-web" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155245 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155251 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155261 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="alertmanager" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155267 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="alertmanager" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155274 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="config-reloader" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155279 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="config-reloader" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155285 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="prom-label-proxy" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155292 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="prom-label-proxy" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155303 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ac7e6f8-46c6-400f-b4bf-ab4749aba609" containerName="console" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155308 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ac7e6f8-46c6-400f-b4bf-ab4749aba609" containerName="console" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155316 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="init-config-reloader" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155322 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="init-config-reloader" Jul 24 13:41:07.155327 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155328 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" containerName="registry" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155334 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" containerName="registry" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155341 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-metric" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155347 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-metric" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155393 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="30ccbce5-4a45-4a1c-8b3f-ce069641e8e3" containerName="registry" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155400 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-web" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155407 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155412 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="prom-label-proxy" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155418 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="config-reloader" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155423 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="kube-rbac-proxy-metric" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155428 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ac7e6f8-46c6-400f-b4bf-ab4749aba609" containerName="console" Jul 24 13:41:07.155694 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.155434 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" containerName="alertmanager" Jul 24 13:41:07.158917 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.158887 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.161588 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161565 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 13:41:07.161823 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161801 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 13:41:07.161895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161850 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 13:41:07.161895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161864 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 13:41:07.161895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161880 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 13:41:07.162023 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161921 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 13:41:07.162023 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.161869 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-twmzz\"" Jul 24 13:41:07.162119 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.162069 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 13:41:07.162190 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.162172 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 13:41:07.167193 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.167171 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 13:41:07.171706 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.171670 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:41:07.293988 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.293952 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294032 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-web-config\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-config-out\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5z9h\" (UniqueName: \"kubernetes.io/projected/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-kube-api-access-k5z9h\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.294352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.294341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395187 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395247 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-web-config\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395385 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-config-out\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5z9h\" (UniqueName: \"kubernetes.io/projected/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-kube-api-access-k5z9h\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.395834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.395615 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.396475 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.396132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.396608 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.396548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398474 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398444 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-web-config\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398595 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398473 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398595 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398506 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398696 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398584 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398813 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398954 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-config-out\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.398954 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.398881 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.399267 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.399251 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.400191 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.400174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.406824 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.406802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5z9h\" (UniqueName: \"kubernetes.io/projected/bd9f2df5-bd7d-43c9-8751-0e3ced766e3f-kube-api-access-k5z9h\") pod \"alertmanager-main-0\" (UID: \"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.469273 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.469232 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 13:41:07.631776 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.631745 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 13:41:07.633767 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:41:07.633705 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9f2df5_bd7d_43c9_8751_0e3ced766e3f.slice/crio-577b885751b2a72e920ed2e7cce080d7e95e89e1f083fe041e4bf10218c2329c WatchSource:0}: Error finding container 577b885751b2a72e920ed2e7cce080d7e95e89e1f083fe041e4bf10218c2329c: Status 404 returned error can't find the container with id 577b885751b2a72e920ed2e7cce080d7e95e89e1f083fe041e4bf10218c2329c Jul 24 13:41:07.797466 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.797421 2568 generic.go:356] "Generic (PLEG): container finished" podID="bd9f2df5-bd7d-43c9-8751-0e3ced766e3f" containerID="7ed9f5723968cd4e8beb6cf35a5788702e8f21150c4cab5a51ccf306e0ff16b3" exitCode=0 Jul 24 13:41:07.797940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.797466 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerDied","Data":"7ed9f5723968cd4e8beb6cf35a5788702e8f21150c4cab5a51ccf306e0ff16b3"} Jul 24 13:41:07.797940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.797535 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"577b885751b2a72e920ed2e7cce080d7e95e89e1f083fe041e4bf10218c2329c"} Jul 24 13:41:07.859790 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:07.859754 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86620919-a5dc-4674-a6cf-4cee43cdccda" path="/var/lib/kubelet/pods/86620919-a5dc-4674-a6cf-4cee43cdccda/volumes" Jul 24 13:41:08.806005 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.805919 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"0e8a8ce3214d1b072aeffe35b6fcdb18cda80b63c1164228b93af743481e3b19"} Jul 24 13:41:08.806005 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.805983 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"c8de5cde62417d2a9af87e1a8b056b8f81c10a7ce485ee10decd19a7389ead10"} Jul 24 13:41:08.806469 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.806010 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"399f9f8309e9e71fcc727abeee05306644c8c33a7eeb1512832a2947f588f2c1"} Jul 24 13:41:08.806469 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.806036 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"fcd641d46e262c9adc527be15809dd2906af05de7d00d0eb52b13be765f06930"} Jul 24 13:41:08.806469 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.806059 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"966cdbffa3923d2e54acedb68dbbb2939186e8f8f4e81d8f34f40cac49d1f52c"} Jul 24 13:41:08.806469 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.806082 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd9f2df5-bd7d-43c9-8751-0e3ced766e3f","Type":"ContainerStarted","Data":"ccd28f449fab18dbc3bc1d78ab368726042097e37dcbd328e4c26963ec9147b7"} Jul 24 13:41:08.835589 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:08.835534 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.814394585 podStartE2EDuration="1.835514416s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.119831ms" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:08.832907986 +0000 UTC m=+165.443387732" watchObservedRunningTime="2026-07-24 13:41:08.835514416 +0000 UTC m=+165.445994162" Jul 24 13:41:09.452913 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.452877 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5fb74759f-dfw7t"] Jul 24 13:41:09.455230 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.455207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.458023 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.457994 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4lg9w\"" Jul 24 13:41:09.458157 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.458030 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 13:41:09.458157 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.458062 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 13:41:09.458277 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.458239 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 13:41:09.458335 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.458300 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 13:41:09.458392 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.458340 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 13:41:09.464398 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.464376 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 13:41:09.468130 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.468108 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fb74759f-dfw7t"] Jul 24 13:41:09.614934 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.614901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-secret-telemeter-client\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.614934 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.614941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-federate-client-tls\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.615151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.614967 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-metrics-client-ca\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.615151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.615030 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbq44\" (UniqueName: \"kubernetes.io/projected/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-kube-api-access-kbq44\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.615151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.615070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-telemeter-client-tls\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.615151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.615084 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.615151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.615118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.615305 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.615191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-serving-certs-ca-bundle\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716538 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716454 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-serving-certs-ca-bundle\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716538 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-secret-telemeter-client\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716681 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716626 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-federate-client-tls\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716681 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-metrics-client-ca\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716805 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbq44\" (UniqueName: \"kubernetes.io/projected/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-kube-api-access-kbq44\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716889 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716860 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-telemeter-client-tls\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.716950 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.717004 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.716969 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.717278 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.717250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-serving-certs-ca-bundle\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.717433 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.717412 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-metrics-client-ca\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.718253 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.718229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.719405 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.719371 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.719527 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.719509 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-secret-telemeter-client\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.719648 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.719608 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-telemeter-client-tls\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.719699 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.719666 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-federate-client-tls\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.724677 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.724656 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbq44\" (UniqueName: \"kubernetes.io/projected/4421bf75-f6c0-4329-96d6-3ac88c0fa63b-kube-api-access-kbq44\") pod \"telemeter-client-5fb74759f-dfw7t\" (UID: \"4421bf75-f6c0-4329-96d6-3ac88c0fa63b\") " pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.765192 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.765141 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" Jul 24 13:41:09.923046 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:09.923015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fb74759f-dfw7t"] Jul 24 13:41:09.925563 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:41:09.925538 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4421bf75_f6c0_4329_96d6_3ac88c0fa63b.slice/crio-5c81bbeaac232199cfe3a372648f750252ce3fcbd6dd06ec82bc0e8cc0cb6dc5 WatchSource:0}: Error finding container 5c81bbeaac232199cfe3a372648f750252ce3fcbd6dd06ec82bc0e8cc0cb6dc5: Status 404 returned error can't find the container with id 5c81bbeaac232199cfe3a372648f750252ce3fcbd6dd06ec82bc0e8cc0cb6dc5 Jul 24 13:41:10.818207 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:10.818169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" event={"ID":"4421bf75-f6c0-4329-96d6-3ac88c0fa63b","Type":"ContainerStarted","Data":"5c81bbeaac232199cfe3a372648f750252ce3fcbd6dd06ec82bc0e8cc0cb6dc5"} Jul 24 13:41:11.824119 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:11.824084 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" event={"ID":"4421bf75-f6c0-4329-96d6-3ac88c0fa63b","Type":"ContainerStarted","Data":"ee19809a0e2f0fbc26ee2ead018293e4929ca71e16784a4a050412e127fa53fa"} Jul 24 13:41:11.824584 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:11.824146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" event={"ID":"4421bf75-f6c0-4329-96d6-3ac88c0fa63b","Type":"ContainerStarted","Data":"538afd6631b3518dc8ab86ac81aea97c123c992d4a5c35e1bfaf02892fee5916"} Jul 24 13:41:11.824584 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:11.824167 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" event={"ID":"4421bf75-f6c0-4329-96d6-3ac88c0fa63b","Type":"ContainerStarted","Data":"ca370733ae6e9c76ca5b9e32e1ce2cbec58ac74d5759211364aec8aad4c4fa35"} Jul 24 13:41:11.855252 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:11.855137 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5fb74759f-dfw7t" podStartSLOduration=1.200487666 podStartE2EDuration="2.855120019s" totalImagesPullingTime="1.654632353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:11.848600851 +0000 UTC m=+168.459080598" watchObservedRunningTime="2026-07-24 13:41:11.855120019 +0000 UTC m=+168.465599765" Jul 24 13:41:12.491628 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.491590 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5796f84fd7-9vzvb"] Jul 24 13:41:12.494156 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.494126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.506276 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.506246 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5796f84fd7-9vzvb"] Jul 24 13:41:12.648645 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648602 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhrt\" (UniqueName: \"kubernetes.io/projected/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-kube-api-access-tvhrt\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.648846 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-serving-cert\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.648846 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-oauth-config\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.648846 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648818 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-service-ca\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.649012 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648850 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-config\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.649012 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-oauth-serving-cert\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.649012 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.648907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-trusted-ca-bundle\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750337 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-serving-cert\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750337 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-oauth-config\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750337 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-service-ca\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750614 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-config\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750614 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-oauth-serving-cert\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750614 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-trusted-ca-bundle\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.750614 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.750482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhrt\" (UniqueName: \"kubernetes.io/projected/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-kube-api-access-tvhrt\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.751274 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.751243 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-oauth-serving-cert\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.751414 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.751246 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-service-ca\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.751414 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.751336 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-config\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.751527 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.751415 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-trusted-ca-bundle\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.752961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.752933 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-oauth-config\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.752961 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.752952 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-serving-cert\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.758484 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.758461 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhrt\" (UniqueName: \"kubernetes.io/projected/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-kube-api-access-tvhrt\") pod \"console-5796f84fd7-9vzvb\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.805587 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.805552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:12.948346 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:12.948319 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5796f84fd7-9vzvb"] Jul 24 13:41:12.951185 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:41:12.951154 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf68c87_e00c_4214_9d42_5d2b26bd8a40.slice/crio-04b8a8be1065a9064d1a850ca77f98861813110c8b6338caebd5bd82aa8d6dad WatchSource:0}: Error finding container 04b8a8be1065a9064d1a850ca77f98861813110c8b6338caebd5bd82aa8d6dad: Status 404 returned error can't find the container with id 04b8a8be1065a9064d1a850ca77f98861813110c8b6338caebd5bd82aa8d6dad Jul 24 13:41:13.836222 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:13.836183 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796f84fd7-9vzvb" event={"ID":"5bf68c87-e00c-4214-9d42-5d2b26bd8a40","Type":"ContainerStarted","Data":"5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7"} Jul 24 13:41:13.836403 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:13.836247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796f84fd7-9vzvb" event={"ID":"5bf68c87-e00c-4214-9d42-5d2b26bd8a40","Type":"ContainerStarted","Data":"04b8a8be1065a9064d1a850ca77f98861813110c8b6338caebd5bd82aa8d6dad"} Jul 24 13:41:13.853322 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:13.853273 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5796f84fd7-9vzvb" podStartSLOduration=1.8532564759999999 podStartE2EDuration="1.853256476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:41:13.852433465 +0000 UTC m=+170.462913212" watchObservedRunningTime="2026-07-24 13:41:13.853256476 +0000 UTC m=+170.463736222" Jul 24 13:41:22.806635 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:22.806592 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:22.807136 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:22.806653 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:22.811580 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:22.811553 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:22.877252 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:22.877221 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:41:22.925517 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:22.925483 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b969ddcd9-8rnjl"] Jul 24 13:41:47.947364 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:47.947319 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b969ddcd9-8rnjl" podUID="9db948d2-4b5b-435c-8875-49b2e9979130" containerName="console" containerID="cri-o://a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646" gracePeriod=15 Jul 24 13:41:48.192289 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.192263 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:41:48.243676 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243596 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-console-config\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.243834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243677 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-oauth-serving-cert\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.243834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243714 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc2xf\" (UniqueName: \"kubernetes.io/projected/9db948d2-4b5b-435c-8875-49b2e9979130-kube-api-access-zc2xf\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.243834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243800 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-oauth-config\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.243948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243839 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-trusted-ca-bundle\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.243948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243899 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-serving-cert\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.243948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.243923 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-service-ca\") pod \"9db948d2-4b5b-435c-8875-49b2e9979130\" (UID: \"9db948d2-4b5b-435c-8875-49b2e9979130\") " Jul 24 13:41:48.244100 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.244039 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-console-config" (OuterVolumeSpecName: "console-config") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:48.244153 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.244109 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:48.244331 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.244306 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-console-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.244476 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.244338 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-oauth-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.244476 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.244396 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:48.244476 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.244404 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-service-ca" (OuterVolumeSpecName: "service-ca") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:41:48.246054 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.246023 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db948d2-4b5b-435c-8875-49b2e9979130-kube-api-access-zc2xf" (OuterVolumeSpecName: "kube-api-access-zc2xf") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "kube-api-access-zc2xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:41:48.246197 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.246172 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:48.246256 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.246186 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9db948d2-4b5b-435c-8875-49b2e9979130" (UID: "9db948d2-4b5b-435c-8875-49b2e9979130"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:41:48.344948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.344903 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-oauth-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.344948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.344939 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-trusted-ca-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.344948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.344949 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9db948d2-4b5b-435c-8875-49b2e9979130-console-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.344948 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.344959 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9db948d2-4b5b-435c-8875-49b2e9979130-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.345208 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.344968 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2xf\" (UniqueName: \"kubernetes.io/projected/9db948d2-4b5b-435c-8875-49b2e9979130-kube-api-access-zc2xf\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:41:48.980059 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.980022 2568 generic.go:356] "Generic (PLEG): container finished" podID="9db948d2-4b5b-435c-8875-49b2e9979130" containerID="a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646" exitCode=0 Jul 24 13:41:48.980485 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.980073 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b969ddcd9-8rnjl" Jul 24 13:41:48.980485 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.980114 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b969ddcd9-8rnjl" event={"ID":"9db948d2-4b5b-435c-8875-49b2e9979130","Type":"ContainerDied","Data":"a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646"} Jul 24 13:41:48.980485 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.980169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b969ddcd9-8rnjl" event={"ID":"9db948d2-4b5b-435c-8875-49b2e9979130","Type":"ContainerDied","Data":"d5a3866bdc6020641e01110e74fdc99fb2579de97539e26b963bbfcde0698171"} Jul 24 13:41:48.980485 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.980195 2568 scope.go:117] "RemoveContainer" containerID="a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646" Jul 24 13:41:48.989328 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.989309 2568 scope.go:117] "RemoveContainer" containerID="a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646" Jul 24 13:41:48.989611 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:41:48.989591 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646\": container with ID starting with a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646 not found: ID does not exist" containerID="a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646" Jul 24 13:41:48.989650 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:48.989622 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646"} err="failed to get container status \"a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646\": rpc error: code = NotFound desc = could not find container \"a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646\": container with ID starting with a0180767a438bba16d8f151a76d83a9b9f9014c7af46311758a803527d2bd646 not found: ID does not exist" Jul 24 13:41:49.000910 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:49.000868 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b969ddcd9-8rnjl"] Jul 24 13:41:49.004995 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:49.004969 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b969ddcd9-8rnjl"] Jul 24 13:41:49.858187 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:41:49.858155 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9db948d2-4b5b-435c-8875-49b2e9979130" path="/var/lib/kubelet/pods/9db948d2-4b5b-435c-8875-49b2e9979130/volumes" Jul 24 13:42:22.102788 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.102750 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c47b44689-nrffw"] Jul 24 13:42:22.103338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.103081 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9db948d2-4b5b-435c-8875-49b2e9979130" containerName="console" Jul 24 13:42:22.103338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.103094 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9db948d2-4b5b-435c-8875-49b2e9979130" containerName="console" Jul 24 13:42:22.103338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.103154 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9db948d2-4b5b-435c-8875-49b2e9979130" containerName="console" Jul 24 13:42:22.106285 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.106261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.119010 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.118979 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c47b44689-nrffw"] Jul 24 13:42:22.140075 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-serving-cert\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.140259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-oauth-config\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.140259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-oauth-serving-cert\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.140259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140211 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-config\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.140259 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25jd\" (UniqueName: \"kubernetes.io/projected/da2119de-dbde-4560-9e8a-7ecaebad0afb-kube-api-access-r25jd\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.140418 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-trusted-ca-bundle\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.140418 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.140395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-service-ca\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.240866 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.240831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-config\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.240866 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.240873 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25jd\" (UniqueName: \"kubernetes.io/projected/da2119de-dbde-4560-9e8a-7ecaebad0afb-kube-api-access-r25jd\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241123 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.240901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-trusted-ca-bundle\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241123 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.240929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-service-ca\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241123 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.240975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-serving-cert\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241271 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.241115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-oauth-config\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241271 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.241178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-oauth-serving-cert\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241646 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.241618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-config\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241802 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.241770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-service-ca\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.241854 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.241841 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-oauth-serving-cert\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.242225 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.242208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-trusted-ca-bundle\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.243403 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.243374 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-oauth-config\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.243624 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.243604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-serving-cert\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.249378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.249352 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25jd\" (UniqueName: \"kubernetes.io/projected/da2119de-dbde-4560-9e8a-7ecaebad0afb-kube-api-access-r25jd\") pod \"console-7c47b44689-nrffw\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.417954 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.417922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:22.557605 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:22.557455 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c47b44689-nrffw"] Jul 24 13:42:22.560493 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:42:22.560464 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2119de_dbde_4560_9e8a_7ecaebad0afb.slice/crio-8de98ecd92709589fd69d3c4151326e7867682a5b1c7ff361e2a43fc3e0aeb6d WatchSource:0}: Error finding container 8de98ecd92709589fd69d3c4151326e7867682a5b1c7ff361e2a43fc3e0aeb6d: Status 404 returned error can't find the container with id 8de98ecd92709589fd69d3c4151326e7867682a5b1c7ff361e2a43fc3e0aeb6d Jul 24 13:42:23.122906 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:23.122869 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47b44689-nrffw" event={"ID":"da2119de-dbde-4560-9e8a-7ecaebad0afb","Type":"ContainerStarted","Data":"e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a"} Jul 24 13:42:23.123285 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:23.122924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47b44689-nrffw" event={"ID":"da2119de-dbde-4560-9e8a-7ecaebad0afb","Type":"ContainerStarted","Data":"8de98ecd92709589fd69d3c4151326e7867682a5b1c7ff361e2a43fc3e0aeb6d"} Jul 24 13:42:23.141354 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:23.141303 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c47b44689-nrffw" podStartSLOduration=1.141286877 podStartE2EDuration="1.141286877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:42:23.140642769 +0000 UTC m=+239.751122515" watchObservedRunningTime="2026-07-24 13:42:23.141286877 +0000 UTC m=+239.751766622" Jul 24 13:42:32.418064 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:32.418024 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:32.418483 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:32.418098 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:32.422903 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:32.422881 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:33.166743 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:33.166692 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:42:33.216534 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:33.216498 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5796f84fd7-9vzvb"] Jul 24 13:42:58.238812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.238701 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5796f84fd7-9vzvb" podUID="5bf68c87-e00c-4214-9d42-5d2b26bd8a40" containerName="console" containerID="cri-o://5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7" gracePeriod=15 Jul 24 13:42:58.489746 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.489654 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:42:58.661649 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661609 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-oauth-serving-cert\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.661873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661665 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-service-ca\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.661873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661691 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-oauth-config\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.661873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661715 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvhrt\" (UniqueName: \"kubernetes.io/projected/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-kube-api-access-tvhrt\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.661873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661804 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-serving-cert\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.661873 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661833 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-trusted-ca-bundle\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.662145 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.661883 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-config\") pod \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\" (UID: \"5bf68c87-e00c-4214-9d42-5d2b26bd8a40\") " Jul 24 13:42:58.662145 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.662114 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:42:58.662247 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.662141 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-service-ca" (OuterVolumeSpecName: "service-ca") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:42:58.662410 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.662377 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-config" (OuterVolumeSpecName: "console-config") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:42:58.662533 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.662452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:42:58.664016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.663991 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-kube-api-access-tvhrt" (OuterVolumeSpecName: "kube-api-access-tvhrt") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "kube-api-access-tvhrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:42:58.664085 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.664035 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:42:58.664132 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.664074 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5bf68c87-e00c-4214-9d42-5d2b26bd8a40" (UID: "5bf68c87-e00c-4214-9d42-5d2b26bd8a40"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763418 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763447 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-oauth-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763457 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763467 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-oauth-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763475 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvhrt\" (UniqueName: \"kubernetes.io/projected/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-kube-api-access-tvhrt\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763484 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-console-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:58.763505 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:58.763491 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bf68c87-e00c-4214-9d42-5d2b26bd8a40-trusted-ca-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:42:59.269623 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.269591 2568 generic.go:356] "Generic (PLEG): container finished" podID="5bf68c87-e00c-4214-9d42-5d2b26bd8a40" containerID="5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7" exitCode=0 Jul 24 13:42:59.270150 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.269639 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5796f84fd7-9vzvb" Jul 24 13:42:59.270150 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.269676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796f84fd7-9vzvb" event={"ID":"5bf68c87-e00c-4214-9d42-5d2b26bd8a40","Type":"ContainerDied","Data":"5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7"} Jul 24 13:42:59.270150 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.269718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5796f84fd7-9vzvb" event={"ID":"5bf68c87-e00c-4214-9d42-5d2b26bd8a40","Type":"ContainerDied","Data":"04b8a8be1065a9064d1a850ca77f98861813110c8b6338caebd5bd82aa8d6dad"} Jul 24 13:42:59.270150 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.269750 2568 scope.go:117] "RemoveContainer" containerID="5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7" Jul 24 13:42:59.279356 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.279328 2568 scope.go:117] "RemoveContainer" containerID="5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7" Jul 24 13:42:59.279768 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:42:59.279718 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7\": container with ID starting with 5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7 not found: ID does not exist" containerID="5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7" Jul 24 13:42:59.279879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.279779 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7"} err="failed to get container status \"5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7\": rpc error: code = NotFound desc = could not find container \"5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7\": container with ID starting with 5682c647698defb4f803638e175652f11b936ff28c12a0da1121365519b71ae7 not found: ID does not exist" Jul 24 13:42:59.296968 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.296926 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5796f84fd7-9vzvb"] Jul 24 13:42:59.299953 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.299920 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5796f84fd7-9vzvb"] Jul 24 13:42:59.858653 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:42:59.858615 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf68c87-e00c-4214-9d42-5d2b26bd8a40" path="/var/lib/kubelet/pods/5bf68c87-e00c-4214-9d42-5d2b26bd8a40/volumes" Jul 24 13:43:23.762068 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:23.762039 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 13:43:33.381586 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.381545 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n"] Jul 24 13:43:33.383939 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.381895 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf68c87-e00c-4214-9d42-5d2b26bd8a40" containerName="console" Jul 24 13:43:33.383939 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.381907 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf68c87-e00c-4214-9d42-5d2b26bd8a40" containerName="console" Jul 24 13:43:33.383939 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.381958 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf68c87-e00c-4214-9d42-5d2b26bd8a40" containerName="console" Jul 24 13:43:33.384922 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.384905 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.387604 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.387576 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 13:43:33.387739 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.387606 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r2xt7\"" Jul 24 13:43:33.387739 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.387617 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 13:43:33.395711 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.395686 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n"] Jul 24 13:43:33.452629 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.452587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.452629 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.452633 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxfd\" (UniqueName: \"kubernetes.io/projected/0fa7a356-ec57-4764-86f8-275c537417a6-kube-api-access-kdxfd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.452886 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.452677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.553884 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.553840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.553884 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.553892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxfd\" (UniqueName: \"kubernetes.io/projected/0fa7a356-ec57-4764-86f8-275c537417a6-kube-api-access-kdxfd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.554147 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.553950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.554235 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.554215 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.554289 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.554264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.563657 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.563631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxfd\" (UniqueName: \"kubernetes.io/projected/0fa7a356-ec57-4764-86f8-275c537417a6-kube-api-access-kdxfd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.696928 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.696835 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:33.836143 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.836116 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n"] Jul 24 13:43:33.838606 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:43:33.838579 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa7a356_ec57_4764_86f8_275c537417a6.slice/crio-46581bb9fb3ebbc088b94224886ceb14dac58d348fd0b193227180571593a0a1 WatchSource:0}: Error finding container 46581bb9fb3ebbc088b94224886ceb14dac58d348fd0b193227180571593a0a1: Status 404 returned error can't find the container with id 46581bb9fb3ebbc088b94224886ceb14dac58d348fd0b193227180571593a0a1 Jul 24 13:43:33.840487 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:33.840471 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:43:34.419258 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:34.419220 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" event={"ID":"0fa7a356-ec57-4764-86f8-275c537417a6","Type":"ContainerStarted","Data":"46581bb9fb3ebbc088b94224886ceb14dac58d348fd0b193227180571593a0a1"} Jul 24 13:43:40.445041 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:40.445002 2568 generic.go:356] "Generic (PLEG): container finished" podID="0fa7a356-ec57-4764-86f8-275c537417a6" containerID="9da55e1b8c9cfa73104c7765c799bbc4868afad8ba5ede2c5165050b1d707e70" exitCode=0 Jul 24 13:43:40.445463 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:40.445091 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" event={"ID":"0fa7a356-ec57-4764-86f8-275c537417a6","Type":"ContainerDied","Data":"9da55e1b8c9cfa73104c7765c799bbc4868afad8ba5ede2c5165050b1d707e70"} Jul 24 13:43:43.459742 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:43.459701 2568 generic.go:356] "Generic (PLEG): container finished" podID="0fa7a356-ec57-4764-86f8-275c537417a6" containerID="a9043c210e7214be2b486d953ce082f7bf352a096963a5edb979957d66d181e8" exitCode=0 Jul 24 13:43:43.460231 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:43.459794 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" event={"ID":"0fa7a356-ec57-4764-86f8-275c537417a6","Type":"ContainerDied","Data":"a9043c210e7214be2b486d953ce082f7bf352a096963a5edb979957d66d181e8"} Jul 24 13:43:50.503301 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:50.503219 2568 generic.go:356] "Generic (PLEG): container finished" podID="0fa7a356-ec57-4764-86f8-275c537417a6" containerID="a8c697d4eab8a0275ad6ac2af7e39424db9cbcf8919747e022e79ffe4cbc5702" exitCode=0 Jul 24 13:43:50.503301 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:50.503282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" event={"ID":"0fa7a356-ec57-4764-86f8-275c537417a6","Type":"ContainerDied","Data":"a8c697d4eab8a0275ad6ac2af7e39424db9cbcf8919747e022e79ffe4cbc5702"} Jul 24 13:43:51.637814 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.637787 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:51.716703 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.716663 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdxfd\" (UniqueName: \"kubernetes.io/projected/0fa7a356-ec57-4764-86f8-275c537417a6-kube-api-access-kdxfd\") pod \"0fa7a356-ec57-4764-86f8-275c537417a6\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " Jul 24 13:43:51.716929 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.716720 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-util\") pod \"0fa7a356-ec57-4764-86f8-275c537417a6\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " Jul 24 13:43:51.716929 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.716775 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-bundle\") pod \"0fa7a356-ec57-4764-86f8-275c537417a6\" (UID: \"0fa7a356-ec57-4764-86f8-275c537417a6\") " Jul 24 13:43:51.717477 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.717452 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-bundle" (OuterVolumeSpecName: "bundle") pod "0fa7a356-ec57-4764-86f8-275c537417a6" (UID: "0fa7a356-ec57-4764-86f8-275c537417a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:43:51.718955 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.718928 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa7a356-ec57-4764-86f8-275c537417a6-kube-api-access-kdxfd" (OuterVolumeSpecName: "kube-api-access-kdxfd") pod "0fa7a356-ec57-4764-86f8-275c537417a6" (UID: "0fa7a356-ec57-4764-86f8-275c537417a6"). InnerVolumeSpecName "kube-api-access-kdxfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:43:51.721412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.721389 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-util" (OuterVolumeSpecName: "util") pod "0fa7a356-ec57-4764-86f8-275c537417a6" (UID: "0fa7a356-ec57-4764-86f8-275c537417a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:43:51.818140 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.818039 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdxfd\" (UniqueName: \"kubernetes.io/projected/0fa7a356-ec57-4764-86f8-275c537417a6-kube-api-access-kdxfd\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:43:51.818140 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.818076 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:43:51.818140 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:51.818091 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fa7a356-ec57-4764-86f8-275c537417a6-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:43:52.513350 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:52.513314 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" event={"ID":"0fa7a356-ec57-4764-86f8-275c537417a6","Type":"ContainerDied","Data":"46581bb9fb3ebbc088b94224886ceb14dac58d348fd0b193227180571593a0a1"} Jul 24 13:43:52.513503 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:52.513367 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46581bb9fb3ebbc088b94224886ceb14dac58d348fd0b193227180571593a0a1" Jul 24 13:43:52.513503 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:52.513335 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q266n" Jul 24 13:43:56.284756 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.284704 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw"] Jul 24 13:43:56.285128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285043 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="util" Jul 24 13:43:56.285128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285056 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="util" Jul 24 13:43:56.285128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285067 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="pull" Jul 24 13:43:56.285128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285072 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="pull" Jul 24 13:43:56.285128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285088 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="extract" Jul 24 13:43:56.285128 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285093 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="extract" Jul 24 13:43:56.285323 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.285150 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fa7a356-ec57-4764-86f8-275c537417a6" containerName="extract" Jul 24 13:43:56.329422 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.329380 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw"] Jul 24 13:43:56.329587 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.329488 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.332172 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.332149 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ztkrt\"" Jul 24 13:43:56.332310 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.332215 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:43:56.332310 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.332228 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 13:43:56.460265 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.460224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwgn\" (UniqueName: \"kubernetes.io/projected/69c20ffb-3a7a-4222-8a58-695443dc73d2-kube-api-access-qqwgn\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vrgw\" (UID: \"69c20ffb-3a7a-4222-8a58-695443dc73d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.460453 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.460341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c20ffb-3a7a-4222-8a58-695443dc73d2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vrgw\" (UID: \"69c20ffb-3a7a-4222-8a58-695443dc73d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.561790 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.561673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwgn\" (UniqueName: \"kubernetes.io/projected/69c20ffb-3a7a-4222-8a58-695443dc73d2-kube-api-access-qqwgn\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vrgw\" (UID: \"69c20ffb-3a7a-4222-8a58-695443dc73d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.561790 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.561789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c20ffb-3a7a-4222-8a58-695443dc73d2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vrgw\" (UID: \"69c20ffb-3a7a-4222-8a58-695443dc73d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.562199 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.562179 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c20ffb-3a7a-4222-8a58-695443dc73d2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vrgw\" (UID: \"69c20ffb-3a7a-4222-8a58-695443dc73d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.572541 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.572514 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwgn\" (UniqueName: \"kubernetes.io/projected/69c20ffb-3a7a-4222-8a58-695443dc73d2-kube-api-access-qqwgn\") pod \"cert-manager-operator-controller-manager-55c444bf86-7vrgw\" (UID: \"69c20ffb-3a7a-4222-8a58-695443dc73d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.639483 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.639452 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" Jul 24 13:43:56.812027 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:56.811958 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw"] Jul 24 13:43:56.814991 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:43:56.814958 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c20ffb_3a7a_4222_8a58_695443dc73d2.slice/crio-d16e00bde3d99e822e32b408e802eb3ff19febf1bbd40b90b79c3fd2143c51a4 WatchSource:0}: Error finding container d16e00bde3d99e822e32b408e802eb3ff19febf1bbd40b90b79c3fd2143c51a4: Status 404 returned error can't find the container with id d16e00bde3d99e822e32b408e802eb3ff19febf1bbd40b90b79c3fd2143c51a4 Jul 24 13:43:57.535626 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:57.535592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" event={"ID":"69c20ffb-3a7a-4222-8a58-695443dc73d2","Type":"ContainerStarted","Data":"d16e00bde3d99e822e32b408e802eb3ff19febf1bbd40b90b79c3fd2143c51a4"} Jul 24 13:43:59.549785 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:59.549690 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" event={"ID":"69c20ffb-3a7a-4222-8a58-695443dc73d2","Type":"ContainerStarted","Data":"d3de2d72685b83cac93ca6ec183940c9ff8926f20ccc977067322db709455f78"} Jul 24 13:43:59.573703 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:43:59.573637 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-7vrgw" podStartSLOduration=1.256337815 podStartE2EDuration="3.573615153s" totalImagesPullingTime="2.317277338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:43:59.572155872 +0000 UTC m=+336.182635609" watchObservedRunningTime="2026-07-24 13:43:59.573615153 +0000 UTC m=+336.184094901" Jul 24 13:44:00.828481 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:00.828445 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw"] Jul 24 13:44:00.852117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:00.852086 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw"] Jul 24 13:44:00.852294 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:00.852228 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:00.854831 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:00.854809 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 13:44:00.856008 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:00.855985 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 13:44:00.856076 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:00.856043 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r2xt7\"" Jul 24 13:44:01.001905 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.001873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77w7t\" (UniqueName: \"kubernetes.io/projected/af68395a-f63d-4ec3-8e6b-1a679c992855-kube-api-access-77w7t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.002078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.001928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.002078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.002005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.103055 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.102963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.103055 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.103024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77w7t\" (UniqueName: \"kubernetes.io/projected/af68395a-f63d-4ec3-8e6b-1a679c992855-kube-api-access-77w7t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.103294 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.103087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.103383 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.103365 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.103476 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.103456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.119393 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.119360 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77w7t\" (UniqueName: \"kubernetes.io/projected/af68395a-f63d-4ec3-8e6b-1a679c992855-kube-api-access-77w7t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.163981 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.163943 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:01.317946 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.317919 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw"] Jul 24 13:44:01.320640 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:44:01.320617 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf68395a_f63d_4ec3_8e6b_1a679c992855.slice/crio-969d846e9959146916cbe756f0e8d80556823ee315237e0c1880367629fa12a3 WatchSource:0}: Error finding container 969d846e9959146916cbe756f0e8d80556823ee315237e0c1880367629fa12a3: Status 404 returned error can't find the container with id 969d846e9959146916cbe756f0e8d80556823ee315237e0c1880367629fa12a3 Jul 24 13:44:01.560857 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.560824 2568 generic.go:356] "Generic (PLEG): container finished" podID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerID="c19300ebb93f12d28327d2f5ec8aca98204602d619497779ed719df4f0e8fef1" exitCode=0 Jul 24 13:44:01.561024 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.560906 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" event={"ID":"af68395a-f63d-4ec3-8e6b-1a679c992855","Type":"ContainerDied","Data":"c19300ebb93f12d28327d2f5ec8aca98204602d619497779ed719df4f0e8fef1"} Jul 24 13:44:01.561024 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:01.560964 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" event={"ID":"af68395a-f63d-4ec3-8e6b-1a679c992855","Type":"ContainerStarted","Data":"969d846e9959146916cbe756f0e8d80556823ee315237e0c1880367629fa12a3"} Jul 24 13:44:02.471939 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.471772 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wfcsz"] Jul 24 13:44:02.490213 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.490050 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wfcsz"] Jul 24 13:44:02.490213 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.490173 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.492793 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.492766 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 13:44:02.494041 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.493959 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 13:44:02.494041 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.493968 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-cq4vl\"" Jul 24 13:44:02.615672 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.615633 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3b1e739-de44-4da5-8515-19ea37090235-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wfcsz\" (UID: \"a3b1e739-de44-4da5-8515-19ea37090235\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.615672 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.615676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jznhk\" (UniqueName: \"kubernetes.io/projected/a3b1e739-de44-4da5-8515-19ea37090235-kube-api-access-jznhk\") pod \"cert-manager-webhook-6486ffd997-wfcsz\" (UID: \"a3b1e739-de44-4da5-8515-19ea37090235\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.717016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.716979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3b1e739-de44-4da5-8515-19ea37090235-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wfcsz\" (UID: \"a3b1e739-de44-4da5-8515-19ea37090235\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.717016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.717019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jznhk\" (UniqueName: \"kubernetes.io/projected/a3b1e739-de44-4da5-8515-19ea37090235-kube-api-access-jznhk\") pod \"cert-manager-webhook-6486ffd997-wfcsz\" (UID: \"a3b1e739-de44-4da5-8515-19ea37090235\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.725501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.725423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3b1e739-de44-4da5-8515-19ea37090235-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wfcsz\" (UID: \"a3b1e739-de44-4da5-8515-19ea37090235\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.725653 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.725530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jznhk\" (UniqueName: \"kubernetes.io/projected/a3b1e739-de44-4da5-8515-19ea37090235-kube-api-access-jznhk\") pod \"cert-manager-webhook-6486ffd997-wfcsz\" (UID: \"a3b1e739-de44-4da5-8515-19ea37090235\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.783251 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.783214 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d"] Jul 24 13:44:02.805829 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.805789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d"] Jul 24 13:44:02.806016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.805897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:02.808498 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.808474 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qhxkd\"" Jul 24 13:44:02.814181 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.814152 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:02.918999 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.918963 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0f44f16-cce2-43d6-834c-0650b32ec992-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tzl7d\" (UID: \"b0f44f16-cce2-43d6-834c-0650b32ec992\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:02.919204 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:02.919075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445l9\" (UniqueName: \"kubernetes.io/projected/b0f44f16-cce2-43d6-834c-0650b32ec992-kube-api-access-445l9\") pod \"cert-manager-cainjector-559cdd8c87-tzl7d\" (UID: \"b0f44f16-cce2-43d6-834c-0650b32ec992\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:03.001521 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.001360 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wfcsz"] Jul 24 13:44:03.003938 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:44:03.003906 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3b1e739_de44_4da5_8515_19ea37090235.slice/crio-79d2812334848f9304483283dd1852ba71a359fd7b721250a9b4ea442a1efca2 WatchSource:0}: Error finding container 79d2812334848f9304483283dd1852ba71a359fd7b721250a9b4ea442a1efca2: Status 404 returned error can't find the container with id 79d2812334848f9304483283dd1852ba71a359fd7b721250a9b4ea442a1efca2 Jul 24 13:44:03.020061 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.020030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0f44f16-cce2-43d6-834c-0650b32ec992-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tzl7d\" (UID: \"b0f44f16-cce2-43d6-834c-0650b32ec992\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:03.020188 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.020108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445l9\" (UniqueName: \"kubernetes.io/projected/b0f44f16-cce2-43d6-834c-0650b32ec992-kube-api-access-445l9\") pod \"cert-manager-cainjector-559cdd8c87-tzl7d\" (UID: \"b0f44f16-cce2-43d6-834c-0650b32ec992\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:03.028215 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.028186 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b0f44f16-cce2-43d6-834c-0650b32ec992-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tzl7d\" (UID: \"b0f44f16-cce2-43d6-834c-0650b32ec992\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:03.028433 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.028411 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-445l9\" (UniqueName: \"kubernetes.io/projected/b0f44f16-cce2-43d6-834c-0650b32ec992-kube-api-access-445l9\") pod \"cert-manager-cainjector-559cdd8c87-tzl7d\" (UID: \"b0f44f16-cce2-43d6-834c-0650b32ec992\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:03.118403 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.118369 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" Jul 24 13:44:03.573836 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.573799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" event={"ID":"a3b1e739-de44-4da5-8515-19ea37090235","Type":"ContainerStarted","Data":"79d2812334848f9304483283dd1852ba71a359fd7b721250a9b4ea442a1efca2"} Jul 24 13:44:03.880689 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:03.880559 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d"] Jul 24 13:44:03.883899 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:44:03.883865 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0f44f16_cce2_43d6_834c_0650b32ec992.slice/crio-92b4ac8274acf0f2e4b2f62f74a2e9d9c0993759df6d27e51b6cc75bf7ebdd68 WatchSource:0}: Error finding container 92b4ac8274acf0f2e4b2f62f74a2e9d9c0993759df6d27e51b6cc75bf7ebdd68: Status 404 returned error can't find the container with id 92b4ac8274acf0f2e4b2f62f74a2e9d9c0993759df6d27e51b6cc75bf7ebdd68 Jul 24 13:44:04.582897 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:04.582805 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" event={"ID":"b0f44f16-cce2-43d6-834c-0650b32ec992","Type":"ContainerStarted","Data":"92b4ac8274acf0f2e4b2f62f74a2e9d9c0993759df6d27e51b6cc75bf7ebdd68"} Jul 24 13:44:04.585797 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:04.585762 2568 generic.go:356] "Generic (PLEG): container finished" podID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerID="45e6d948900eb145476c50952162e62657889923ae4887ece7a5871be3e6ed28" exitCode=0 Jul 24 13:44:04.585951 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:04.585825 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" event={"ID":"af68395a-f63d-4ec3-8e6b-1a679c992855","Type":"ContainerDied","Data":"45e6d948900eb145476c50952162e62657889923ae4887ece7a5871be3e6ed28"} Jul 24 13:44:05.596135 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:05.596039 2568 generic.go:356] "Generic (PLEG): container finished" podID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerID="2a7936afc21888f5c008dea1299a58e34e81e538deaaab61bab05292378aeee7" exitCode=0 Jul 24 13:44:05.596621 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:05.596173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" event={"ID":"af68395a-f63d-4ec3-8e6b-1a679c992855","Type":"ContainerDied","Data":"2a7936afc21888f5c008dea1299a58e34e81e538deaaab61bab05292378aeee7"} Jul 24 13:44:06.603589 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.603485 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" event={"ID":"a3b1e739-de44-4da5-8515-19ea37090235","Type":"ContainerStarted","Data":"a465fbcdc23d6bb83f08fdd28c3d31c40077027a505cd04d0dee7b95eb533b68"} Jul 24 13:44:06.603589 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.603545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:06.605281 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.605247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" event={"ID":"b0f44f16-cce2-43d6-834c-0650b32ec992","Type":"ContainerStarted","Data":"4549c5e93dcab208e4754e8f5ba53e0dbc4b7ecc4d6cb8ffa3e0f1857c08f319"} Jul 24 13:44:06.622518 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.622453 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" podStartSLOduration=1.371058347 podStartE2EDuration="4.622432222s" totalImagesPullingTime="3.251373875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:06.621075392 +0000 UTC m=+343.231555185" watchObservedRunningTime="2026-07-24 13:44:06.622432222 +0000 UTC m=+343.232911970" Jul 24 13:44:06.636211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.636151 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tzl7d" podStartSLOduration=2.258166563 podStartE2EDuration="4.636133786s" totalImagesPullingTime="2.377967223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:06.635284203 +0000 UTC m=+343.245763948" watchObservedRunningTime="2026-07-24 13:44:06.636133786 +0000 UTC m=+343.246613532" Jul 24 13:44:06.744427 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.744401 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:06.859338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.859235 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-bundle\") pod \"af68395a-f63d-4ec3-8e6b-1a679c992855\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " Jul 24 13:44:06.859338 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.859285 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77w7t\" (UniqueName: \"kubernetes.io/projected/af68395a-f63d-4ec3-8e6b-1a679c992855-kube-api-access-77w7t\") pod \"af68395a-f63d-4ec3-8e6b-1a679c992855\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " Jul 24 13:44:06.859588 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.859349 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-util\") pod \"af68395a-f63d-4ec3-8e6b-1a679c992855\" (UID: \"af68395a-f63d-4ec3-8e6b-1a679c992855\") " Jul 24 13:44:06.859588 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.859574 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-bundle" (OuterVolumeSpecName: "bundle") pod "af68395a-f63d-4ec3-8e6b-1a679c992855" (UID: "af68395a-f63d-4ec3-8e6b-1a679c992855"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:44:06.861555 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.861519 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af68395a-f63d-4ec3-8e6b-1a679c992855-kube-api-access-77w7t" (OuterVolumeSpecName: "kube-api-access-77w7t") pod "af68395a-f63d-4ec3-8e6b-1a679c992855" (UID: "af68395a-f63d-4ec3-8e6b-1a679c992855"). InnerVolumeSpecName "kube-api-access-77w7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:44:06.863973 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.863912 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-util" (OuterVolumeSpecName: "util") pod "af68395a-f63d-4ec3-8e6b-1a679c992855" (UID: "af68395a-f63d-4ec3-8e6b-1a679c992855"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:44:06.960042 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.959988 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:44:06.960042 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.960038 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af68395a-f63d-4ec3-8e6b-1a679c992855-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:44:06.960042 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:06.960055 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77w7t\" (UniqueName: \"kubernetes.io/projected/af68395a-f63d-4ec3-8e6b-1a679c992855-kube-api-access-77w7t\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:44:07.612078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:07.612047 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" Jul 24 13:44:07.612078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:07.612068 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5l5qw" event={"ID":"af68395a-f63d-4ec3-8e6b-1a679c992855","Type":"ContainerDied","Data":"969d846e9959146916cbe756f0e8d80556823ee315237e0c1880367629fa12a3"} Jul 24 13:44:07.612528 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:07.612152 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969d846e9959146916cbe756f0e8d80556823ee315237e0c1880367629fa12a3" Jul 24 13:44:12.615213 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:12.615178 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wfcsz" Jul 24 13:44:13.407822 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.407784 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8"] Jul 24 13:44:13.408117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408105 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="extract" Jul 24 13:44:13.408165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408118 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="extract" Jul 24 13:44:13.408165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408137 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="pull" Jul 24 13:44:13.408165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408143 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="pull" Jul 24 13:44:13.408165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408156 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="util" Jul 24 13:44:13.408165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408162 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="util" Jul 24 13:44:13.408307 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.408208 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="af68395a-f63d-4ec3-8e6b-1a679c992855" containerName="extract" Jul 24 13:44:13.412199 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.412178 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.416110 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.416087 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 13:44:13.416234 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.416085 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 13:44:13.417079 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.417054 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-bg2sq\"" Jul 24 13:44:13.420029 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.419998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8"] Jul 24 13:44:13.517278 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.517244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3926899b-6fb6-4e21-b4a9-31322ea556f7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mmqh8\" (UID: \"3926899b-6fb6-4e21-b4a9-31322ea556f7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.517471 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.517293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vkcw\" (UniqueName: \"kubernetes.io/projected/3926899b-6fb6-4e21-b4a9-31322ea556f7-kube-api-access-9vkcw\") pod \"openshift-lws-operator-fd8ccff4c-mmqh8\" (UID: \"3926899b-6fb6-4e21-b4a9-31322ea556f7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.618678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.618639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3926899b-6fb6-4e21-b4a9-31322ea556f7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mmqh8\" (UID: \"3926899b-6fb6-4e21-b4a9-31322ea556f7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.619198 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.618693 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vkcw\" (UniqueName: \"kubernetes.io/projected/3926899b-6fb6-4e21-b4a9-31322ea556f7-kube-api-access-9vkcw\") pod \"openshift-lws-operator-fd8ccff4c-mmqh8\" (UID: \"3926899b-6fb6-4e21-b4a9-31322ea556f7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.619198 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.619155 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3926899b-6fb6-4e21-b4a9-31322ea556f7-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mmqh8\" (UID: \"3926899b-6fb6-4e21-b4a9-31322ea556f7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.626684 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.626646 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vkcw\" (UniqueName: \"kubernetes.io/projected/3926899b-6fb6-4e21-b4a9-31322ea556f7-kube-api-access-9vkcw\") pod \"openshift-lws-operator-fd8ccff4c-mmqh8\" (UID: \"3926899b-6fb6-4e21-b4a9-31322ea556f7\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.722095 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.721997 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" Jul 24 13:44:13.860648 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:13.860623 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8"] Jul 24 13:44:13.863096 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:44:13.863067 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3926899b_6fb6_4e21_b4a9_31322ea556f7.slice/crio-208de86a87f8a400b44b8f0eaba55c7359efffef40aca3abef4934e60b444b42 WatchSource:0}: Error finding container 208de86a87f8a400b44b8f0eaba55c7359efffef40aca3abef4934e60b444b42: Status 404 returned error can't find the container with id 208de86a87f8a400b44b8f0eaba55c7359efffef40aca3abef4934e60b444b42 Jul 24 13:44:14.649004 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:14.648968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" event={"ID":"3926899b-6fb6-4e21-b4a9-31322ea556f7","Type":"ContainerStarted","Data":"208de86a87f8a400b44b8f0eaba55c7359efffef40aca3abef4934e60b444b42"} Jul 24 13:44:16.663611 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:16.663564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" event={"ID":"3926899b-6fb6-4e21-b4a9-31322ea556f7","Type":"ContainerStarted","Data":"8028d1b8453021670ef5c66de0813228fea4a0699b4b8670d78f4ce7c5d0ab7f"} Jul 24 13:44:16.680374 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:16.680320 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mmqh8" podStartSLOduration=1.81174596 podStartE2EDuration="3.680303635s" totalImagesPullingTime="1.868557675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:16.679188685 +0000 UTC m=+353.289668435" watchObservedRunningTime="2026-07-24 13:44:16.680303635 +0000 UTC m=+353.290783380" Jul 24 13:44:26.859891 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.859853 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd"] Jul 24 13:44:26.863628 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.863608 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:26.866207 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.866181 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 13:44:26.866305 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.866246 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r2xt7\"" Jul 24 13:44:26.867275 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.867252 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 13:44:26.873452 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.873427 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd"] Jul 24 13:44:26.936589 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.936544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmw4g\" (UniqueName: \"kubernetes.io/projected/675b9635-f276-45b1-a358-77487f33ed84-kube-api-access-jmw4g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:26.936793 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.936603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:26.936793 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:26.936660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.037711 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.037676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmw4g\" (UniqueName: \"kubernetes.io/projected/675b9635-f276-45b1-a358-77487f33ed84-kube-api-access-jmw4g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.037938 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.037719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.037938 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.037761 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.038095 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.038073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.038148 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.038099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.046409 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.046377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmw4g\" (UniqueName: \"kubernetes.io/projected/675b9635-f276-45b1-a358-77487f33ed84-kube-api-access-jmw4g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.173771 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.173712 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:27.319273 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.319249 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd"] Jul 24 13:44:27.320757 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:44:27.320702 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b9635_f276_45b1_a358_77487f33ed84.slice/crio-65836076edb1967efbfad8b049cf73fb3a532707e66d9dced2f935223759df3b WatchSource:0}: Error finding container 65836076edb1967efbfad8b049cf73fb3a532707e66d9dced2f935223759df3b: Status 404 returned error can't find the container with id 65836076edb1967efbfad8b049cf73fb3a532707e66d9dced2f935223759df3b Jul 24 13:44:27.717320 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.717282 2568 generic.go:356] "Generic (PLEG): container finished" podID="675b9635-f276-45b1-a358-77487f33ed84" containerID="89ad7b2ecc6a8d741d200edc04b1b6957969696b4abd7324a4f4756bb903f088" exitCode=0 Jul 24 13:44:27.717498 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.717367 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" event={"ID":"675b9635-f276-45b1-a358-77487f33ed84","Type":"ContainerDied","Data":"89ad7b2ecc6a8d741d200edc04b1b6957969696b4abd7324a4f4756bb903f088"} Jul 24 13:44:27.717498 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:27.717419 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" event={"ID":"675b9635-f276-45b1-a358-77487f33ed84","Type":"ContainerStarted","Data":"65836076edb1967efbfad8b049cf73fb3a532707e66d9dced2f935223759df3b"} Jul 24 13:44:28.723612 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:28.723573 2568 generic.go:356] "Generic (PLEG): container finished" podID="675b9635-f276-45b1-a358-77487f33ed84" containerID="fd9420f69606f5c288b4c65bdfc08caad75123141d7439bc147d9d826aa6f1da" exitCode=0 Jul 24 13:44:28.724038 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:28.723661 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" event={"ID":"675b9635-f276-45b1-a358-77487f33ed84","Type":"ContainerDied","Data":"fd9420f69606f5c288b4c65bdfc08caad75123141d7439bc147d9d826aa6f1da"} Jul 24 13:44:29.730334 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:29.730299 2568 generic.go:356] "Generic (PLEG): container finished" podID="675b9635-f276-45b1-a358-77487f33ed84" containerID="6473244e7455f6145d8c7ab0b1ad49ed45d559056b1b1cc1342340733ce205c5" exitCode=0 Jul 24 13:44:29.730717 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:29.730356 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" event={"ID":"675b9635-f276-45b1-a358-77487f33ed84","Type":"ContainerDied","Data":"6473244e7455f6145d8c7ab0b1ad49ed45d559056b1b1cc1342340733ce205c5"} Jul 24 13:44:30.865419 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.865389 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:30.973541 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.973503 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-util\") pod \"675b9635-f276-45b1-a358-77487f33ed84\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " Jul 24 13:44:30.973768 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.973602 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmw4g\" (UniqueName: \"kubernetes.io/projected/675b9635-f276-45b1-a358-77487f33ed84-kube-api-access-jmw4g\") pod \"675b9635-f276-45b1-a358-77487f33ed84\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " Jul 24 13:44:30.973768 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.973637 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-bundle\") pod \"675b9635-f276-45b1-a358-77487f33ed84\" (UID: \"675b9635-f276-45b1-a358-77487f33ed84\") " Jul 24 13:44:30.974386 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.974344 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-bundle" (OuterVolumeSpecName: "bundle") pod "675b9635-f276-45b1-a358-77487f33ed84" (UID: "675b9635-f276-45b1-a358-77487f33ed84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:44:30.975817 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.975790 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675b9635-f276-45b1-a358-77487f33ed84-kube-api-access-jmw4g" (OuterVolumeSpecName: "kube-api-access-jmw4g") pod "675b9635-f276-45b1-a358-77487f33ed84" (UID: "675b9635-f276-45b1-a358-77487f33ed84"). InnerVolumeSpecName "kube-api-access-jmw4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:44:30.979123 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:30.979092 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-util" (OuterVolumeSpecName: "util") pod "675b9635-f276-45b1-a358-77487f33ed84" (UID: "675b9635-f276-45b1-a358-77487f33ed84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:44:31.075034 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:31.074935 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:44:31.075034 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:31.074972 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmw4g\" (UniqueName: \"kubernetes.io/projected/675b9635-f276-45b1-a358-77487f33ed84-kube-api-access-jmw4g\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:44:31.075034 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:31.074987 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/675b9635-f276-45b1-a358-77487f33ed84-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:44:31.741260 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:31.741219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" event={"ID":"675b9635-f276-45b1-a358-77487f33ed84","Type":"ContainerDied","Data":"65836076edb1967efbfad8b049cf73fb3a532707e66d9dced2f935223759df3b"} Jul 24 13:44:31.741260 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:31.741259 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk76qd" Jul 24 13:44:31.741260 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:31.741270 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65836076edb1967efbfad8b049cf73fb3a532707e66d9dced2f935223759df3b" Jul 24 13:44:39.137360 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137322 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l"] Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137652 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="pull" Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137663 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="pull" Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137676 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="util" Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137681 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="util" Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137694 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="extract" Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137700 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="extract" Jul 24 13:44:39.137879 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.137785 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="675b9635-f276-45b1-a358-77487f33ed84" containerName="extract" Jul 24 13:44:39.144666 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.144635 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.147773 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.147749 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 13:44:39.147931 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.147914 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q2sql\"" Jul 24 13:44:39.148000 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.147951 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 13:44:39.148442 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.148326 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 13:44:39.148633 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.148619 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 13:44:39.164852 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.164818 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l"] Jul 24 13:44:39.250047 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.249995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.250244 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.250065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-tmp\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.250244 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.250120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.250244 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.250207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-manifests\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.250399 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.250284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qng4x\" (UniqueName: \"kubernetes.io/projected/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-kube-api-access-qng4x\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.351018 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.350972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.351018 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.351030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-manifests\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.351264 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.351066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qng4x\" (UniqueName: \"kubernetes.io/projected/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-kube-api-access-qng4x\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.351264 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.351122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.351264 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.351160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-tmp\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.351516 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.351488 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-manifests\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.353612 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.353586 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-tmp\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.353714 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.353647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.353714 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.353689 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.360368 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.360341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qng4x\" (UniqueName: \"kubernetes.io/projected/dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5-kube-api-access-qng4x\") pod \"opendatahub-operator-controller-manager-6c96dc84dd-sps8l\" (UID: \"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5\") " pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.455552 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.455453 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:39.604103 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.604069 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l"] Jul 24 13:44:39.609118 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:44:39.609084 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadf38ad_d85c_4cd7_bbcf_053f7d3df4b5.slice/crio-d1ffe9cbb349ee3bd47214f06008a207c420d4f0f3c6208446b33dd3e6cd7908 WatchSource:0}: Error finding container d1ffe9cbb349ee3bd47214f06008a207c420d4f0f3c6208446b33dd3e6cd7908: Status 404 returned error can't find the container with id d1ffe9cbb349ee3bd47214f06008a207c420d4f0f3c6208446b33dd3e6cd7908 Jul 24 13:44:39.781012 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:39.780924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" event={"ID":"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5","Type":"ContainerStarted","Data":"d1ffe9cbb349ee3bd47214f06008a207c420d4f0f3c6208446b33dd3e6cd7908"} Jul 24 13:44:42.799546 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:42.799507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" event={"ID":"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5","Type":"ContainerStarted","Data":"4ba041509e04b6ec6fd797d4c36268e41f4c24c44e6e1814550783720d506f4e"} Jul 24 13:44:43.806148 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:43.806111 2568 generic.go:356] "Generic (PLEG): container finished" podID="dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5" containerID="4ba041509e04b6ec6fd797d4c36268e41f4c24c44e6e1814550783720d506f4e" exitCode=0 Jul 24 13:44:43.806565 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:43.806209 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" event={"ID":"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5","Type":"ContainerDied","Data":"4ba041509e04b6ec6fd797d4c36268e41f4c24c44e6e1814550783720d506f4e"} Jul 24 13:44:44.813165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:44.813070 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" event={"ID":"dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5","Type":"ContainerStarted","Data":"8f6f75b897119327d0a9bb8d2964bbdcb403cfd33a0c536043728e86bc8717ba"} Jul 24 13:44:44.813538 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:44.813162 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:44.837084 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:44.837034 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" podStartSLOduration=1.266128294 podStartE2EDuration="5.837014601s" totalImagesPullingTime="3.561235922s" totalInitContainerRuntime="1.009650385s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:44:44.836308071 +0000 UTC m=+381.446787818" watchObservedRunningTime="2026-07-24 13:44:44.837014601 +0000 UTC m=+381.447494346" Jul 24 13:44:55.820085 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:55.820047 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c96dc84dd-sps8l" Jul 24 13:44:59.556215 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.556175 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8dpq"] Jul 24 13:44:59.564144 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.564126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:44:59.566909 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.566884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 13:44:59.567167 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.567151 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-ktx2p\"" Jul 24 13:44:59.571323 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.571296 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8dpq"] Jul 24 13:44:59.639168 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.639133 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eccc1867-2a09-47a2-a98c-8d024a1bd00a-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:44:59.639366 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.639181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94vrg\" (UniqueName: \"kubernetes.io/projected/eccc1867-2a09-47a2-a98c-8d024a1bd00a-kube-api-access-94vrg\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:44:59.739945 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.739904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eccc1867-2a09-47a2-a98c-8d024a1bd00a-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:44:59.740149 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.739976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94vrg\" (UniqueName: \"kubernetes.io/projected/eccc1867-2a09-47a2-a98c-8d024a1bd00a-kube-api-access-94vrg\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:44:59.740149 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:44:59.740048 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 13:44:59.740149 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:44:59.740142 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccc1867-2a09-47a2-a98c-8d024a1bd00a-tls-certs podName:eccc1867-2a09-47a2-a98c-8d024a1bd00a nodeName:}" failed. No retries permitted until 2026-07-24 13:45:00.24012564 +0000 UTC m=+396.850605364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/eccc1867-2a09-47a2-a98c-8d024a1bd00a-tls-certs") pod "model-serving-api-54d89bd5fc-v8dpq" (UID: "eccc1867-2a09-47a2-a98c-8d024a1bd00a") : secret "model-serving-api-tls" not found Jul 24 13:44:59.752707 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:44:59.752679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94vrg\" (UniqueName: \"kubernetes.io/projected/eccc1867-2a09-47a2-a98c-8d024a1bd00a-kube-api-access-94vrg\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:45:00.243484 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:00.243446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eccc1867-2a09-47a2-a98c-8d024a1bd00a-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:45:00.245984 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:00.245953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eccc1867-2a09-47a2-a98c-8d024a1bd00a-tls-certs\") pod \"model-serving-api-54d89bd5fc-v8dpq\" (UID: \"eccc1867-2a09-47a2-a98c-8d024a1bd00a\") " pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:45:00.475690 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:00.475637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:45:00.613880 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:00.613847 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v8dpq"] Jul 24 13:45:00.617564 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:45:00.617536 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccc1867_2a09_47a2_a98c_8d024a1bd00a.slice/crio-a3d089ee480f853ac8c4bdd3acefb9746cc8353aab8de44fadbe8bf5b4f72f4b WatchSource:0}: Error finding container a3d089ee480f853ac8c4bdd3acefb9746cc8353aab8de44fadbe8bf5b4f72f4b: Status 404 returned error can't find the container with id a3d089ee480f853ac8c4bdd3acefb9746cc8353aab8de44fadbe8bf5b4f72f4b Jul 24 13:45:00.893180 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:00.893144 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" event={"ID":"eccc1867-2a09-47a2-a98c-8d024a1bd00a","Type":"ContainerStarted","Data":"a3d089ee480f853ac8c4bdd3acefb9746cc8353aab8de44fadbe8bf5b4f72f4b"} Jul 24 13:45:02.905477 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:02.905442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" event={"ID":"eccc1867-2a09-47a2-a98c-8d024a1bd00a","Type":"ContainerStarted","Data":"554a4e5b374b3988de5ba1a9229c38ef80e479984863e7365deca295c89baee8"} Jul 24 13:45:02.905895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:02.905556 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:45:02.925409 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:02.925345 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" podStartSLOduration=2.173247579 podStartE2EDuration="3.925326206s" totalImagesPullingTime="1.752078627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:02.922923506 +0000 UTC m=+399.533403251" watchObservedRunningTime="2026-07-24 13:45:02.925326206 +0000 UTC m=+399.535805951" Jul 24 13:45:06.236248 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.236209 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d"] Jul 24 13:45:06.239735 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.239701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.243410 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.243391 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 13:45:06.243501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.243403 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 13:45:06.243501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.243429 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 13:45:06.243501 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.243463 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-659nq\"" Jul 24 13:45:06.249993 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.249965 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d"] Jul 24 13:45:06.299934 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.299897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a465945-501b-4ccb-b96a-16dfeb24755a-metrics-cert\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.300127 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.299966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a465945-501b-4ccb-b96a-16dfeb24755a-manager-config\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.300127 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.300037 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6lr\" (UniqueName: \"kubernetes.io/projected/8a465945-501b-4ccb-b96a-16dfeb24755a-kube-api-access-rc6lr\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.300127 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.300079 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a465945-501b-4ccb-b96a-16dfeb24755a-cert\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.343422 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.343385 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl"] Jul 24 13:45:06.346783 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.346764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.349599 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349395 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 13:45:06.349599 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349410 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 13:45:06.349599 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349460 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kdb7l\"" Jul 24 13:45:06.349599 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349460 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 13:45:06.349599 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349417 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 13:45:06.349599 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349395 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 13:45:06.349946 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.349756 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 13:45:06.359907 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.359878 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl"] Jul 24 13:45:06.401535 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401497 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3699c26b-4e81-4199-a863-2b6e0011639b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.401742 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a465945-501b-4ccb-b96a-16dfeb24755a-manager-config\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.401742 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.401742 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6lr\" (UniqueName: \"kubernetes.io/projected/8a465945-501b-4ccb-b96a-16dfeb24755a-kube-api-access-rc6lr\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.401742 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrrt\" (UniqueName: \"kubernetes.io/projected/3699c26b-4e81-4199-a863-2b6e0011639b-kube-api-access-vzrrt\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.401924 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.401924 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a465945-501b-4ccb-b96a-16dfeb24755a-cert\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.401924 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3699c26b-4e81-4199-a863-2b6e0011639b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.401924 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.402097 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401929 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a465945-501b-4ccb-b96a-16dfeb24755a-metrics-cert\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.402097 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.401947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3699c26b-4e81-4199-a863-2b6e0011639b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.402321 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.402293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8a465945-501b-4ccb-b96a-16dfeb24755a-manager-config\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.404201 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.404178 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a465945-501b-4ccb-b96a-16dfeb24755a-cert\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.404298 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.404220 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a465945-501b-4ccb-b96a-16dfeb24755a-metrics-cert\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.413693 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.413659 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6lr\" (UniqueName: \"kubernetes.io/projected/8a465945-501b-4ccb-b96a-16dfeb24755a-kube-api-access-rc6lr\") pod \"lws-controller-manager-7f486947b8-4zz5d\" (UID: \"8a465945-501b-4ccb-b96a-16dfeb24755a\") " pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.503154 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3699c26b-4e81-4199-a863-2b6e0011639b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.503154 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.503402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrrt\" (UniqueName: \"kubernetes.io/projected/3699c26b-4e81-4199-a863-2b6e0011639b-kube-api-access-vzrrt\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.503402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.503402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3699c26b-4e81-4199-a863-2b6e0011639b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.503402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.503402 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.503295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3699c26b-4e81-4199-a863-2b6e0011639b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.504067 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.504043 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3699c26b-4e81-4199-a863-2b6e0011639b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.505503 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.505481 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3699c26b-4e81-4199-a863-2b6e0011639b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.505588 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.505545 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.505929 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.505913 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.506048 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.506023 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3699c26b-4e81-4199-a863-2b6e0011639b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.515018 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.514993 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrrt\" (UniqueName: \"kubernetes.io/projected/3699c26b-4e81-4199-a863-2b6e0011639b-kube-api-access-vzrrt\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.515135 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.514997 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3699c26b-4e81-4199-a863-2b6e0011639b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7vtpl\" (UID: \"3699c26b-4e81-4199-a863-2b6e0011639b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.549887 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.549850 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:06.656112 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.656075 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:06.692347 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.692299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d"] Jul 24 13:45:06.699476 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:45:06.699445 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a465945_501b_4ccb_b96a_16dfeb24755a.slice/crio-8f93f983da7a62d1d228c07d2211b0eea1bd32799f43ff5b36d2f5f4d03f847d WatchSource:0}: Error finding container 8f93f983da7a62d1d228c07d2211b0eea1bd32799f43ff5b36d2f5f4d03f847d: Status 404 returned error can't find the container with id 8f93f983da7a62d1d228c07d2211b0eea1bd32799f43ff5b36d2f5f4d03f847d Jul 24 13:45:06.804269 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.804241 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl"] Jul 24 13:45:06.806092 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:45:06.806057 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3699c26b_4e81_4199_a863_2b6e0011639b.slice/crio-a19cf33dbac500e076a87de3cafaeec5fe2cf581871403a4244cfabef1fa1da7 WatchSource:0}: Error finding container a19cf33dbac500e076a87de3cafaeec5fe2cf581871403a4244cfabef1fa1da7: Status 404 returned error can't find the container with id a19cf33dbac500e076a87de3cafaeec5fe2cf581871403a4244cfabef1fa1da7 Jul 24 13:45:06.932344 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.932299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" event={"ID":"3699c26b-4e81-4199-a863-2b6e0011639b","Type":"ContainerStarted","Data":"a19cf33dbac500e076a87de3cafaeec5fe2cf581871403a4244cfabef1fa1da7"} Jul 24 13:45:06.933684 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:06.933657 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" event={"ID":"8a465945-501b-4ccb-b96a-16dfeb24755a","Type":"ContainerStarted","Data":"8f93f983da7a62d1d228c07d2211b0eea1bd32799f43ff5b36d2f5f4d03f847d"} Jul 24 13:45:09.252060 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:09.252016 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 13:45:09.252454 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:09.252098 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 13:45:09.953426 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:09.953390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" event={"ID":"3699c26b-4e81-4199-a863-2b6e0011639b","Type":"ContainerStarted","Data":"7779c9db89b2883c868cc29989a3d005f5a292f6abb4f3da1f33a732332883eb"} Jul 24 13:45:09.953634 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:09.953470 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:09.977884 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:09.977826 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" podStartSLOduration=1.534259399 podStartE2EDuration="3.977803833s" totalImagesPullingTime="2.443544434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:09.975253143 +0000 UTC m=+406.585732890" watchObservedRunningTime="2026-07-24 13:45:09.977803833 +0000 UTC m=+406.588283578" Jul 24 13:45:10.960529 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:10.960490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" event={"ID":"8a465945-501b-4ccb-b96a-16dfeb24755a","Type":"ContainerStarted","Data":"8bc2d9b5355c775a12ed7c6cdde8e4f3f82cc553071c1b51da761dd4a2b4548c"} Jul 24 13:45:10.961102 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:10.960772 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:45:10.962559 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:10.962528 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-7vtpl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 13:45:10.962691 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:10.962587 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" podUID="3699c26b-4e81-4199-a863-2b6e0011639b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 13:45:10.985129 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:10.985075 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" podStartSLOduration=1.546885662 podStartE2EDuration="4.985056274s" totalImagesPullingTime="3.438170612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:45:10.984136299 +0000 UTC m=+407.594616068" watchObservedRunningTime="2026-07-24 13:45:10.985056274 +0000 UTC m=+407.595536020" Jul 24 13:45:13.914646 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:13.914615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-v8dpq" Jul 24 13:45:13.961697 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:13.961668 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7vtpl" Jul 24 13:45:21.967646 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:45:21.967617 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7f486947b8-4zz5d" Jul 24 13:46:14.983888 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:14.983848 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw"] Jul 24 13:46:14.987802 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:14.987779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:14.990544 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:14.990522 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 13:46:14.990678 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:14.990582 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 13:46:14.991520 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:14.991502 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2sl75\"" Jul 24 13:46:14.996847 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:14.996822 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw"] Jul 24 13:46:15.127930 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.127890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.128144 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.127944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.128144 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.128086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-898xs\" (UniqueName: \"kubernetes.io/projected/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-kube-api-access-898xs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.229153 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.229113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-898xs\" (UniqueName: \"kubernetes.io/projected/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-kube-api-access-898xs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.229308 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.229243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.229308 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.229302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.229631 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.229614 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.229669 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.229640 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.238542 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.238482 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-898xs\" (UniqueName: \"kubernetes.io/projected/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-kube-api-access-898xs\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.298658 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.298621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:15.437555 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.437515 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw"] Jul 24 13:46:15.440770 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:46:15.440740 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42e23d2_3ef2_4dc1_ba3e_9aa8c4371c8f.slice/crio-e2b9ce3c7e17e0d836a6139ca178b16e206bf67583f884739a131f868f753394 WatchSource:0}: Error finding container e2b9ce3c7e17e0d836a6139ca178b16e206bf67583f884739a131f868f753394: Status 404 returned error can't find the container with id e2b9ce3c7e17e0d836a6139ca178b16e206bf67583f884739a131f868f753394 Jul 24 13:46:15.783936 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.783838 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph"] Jul 24 13:46:15.787691 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.787665 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:15.795968 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.795943 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph"] Jul 24 13:46:15.934635 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.934587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/4750570c-2073-4de1-a505-7ea27b5d7ff3-kube-api-access-8np2d\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:15.934855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.934649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:15.934855 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:15.934675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.035789 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.035683 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.035789 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.035719 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.036183 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.035840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/4750570c-2073-4de1-a505-7ea27b5d7ff3-kube-api-access-8np2d\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.036183 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.036071 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.036183 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.036147 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.043480 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.043458 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/4750570c-2073-4de1-a505-7ea27b5d7ff3-kube-api-access-8np2d\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.098288 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.098249 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:16.181632 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.181551 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6"] Jul 24 13:46:16.186028 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.186003 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.193611 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.193543 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6"] Jul 24 13:46:16.237653 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.237618 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.237878 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.237670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zxwf\" (UniqueName: \"kubernetes.io/projected/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-kube-api-access-5zxwf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.237878 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.237837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.239963 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.239937 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph"] Jul 24 13:46:16.241417 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:46:16.241392 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4750570c_2073_4de1_a505_7ea27b5d7ff3.slice/crio-5bec695f66490b46d389fba7c4fa833ac0b5c2fde0dc24b85da90eb67e35d29b WatchSource:0}: Error finding container 5bec695f66490b46d389fba7c4fa833ac0b5c2fde0dc24b85da90eb67e35d29b: Status 404 returned error can't find the container with id 5bec695f66490b46d389fba7c4fa833ac0b5c2fde0dc24b85da90eb67e35d29b Jul 24 13:46:16.289834 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.289754 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" event={"ID":"4750570c-2073-4de1-a505-7ea27b5d7ff3","Type":"ContainerStarted","Data":"5bec695f66490b46d389fba7c4fa833ac0b5c2fde0dc24b85da90eb67e35d29b"} Jul 24 13:46:16.291536 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.291507 2568 generic.go:356] "Generic (PLEG): container finished" podID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerID="c3875288d8dc08013a87bebbc1afa6c3d09c05b1e4cbad4e0a2384f8b1949050" exitCode=0 Jul 24 13:46:16.291679 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.291595 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" event={"ID":"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f","Type":"ContainerDied","Data":"c3875288d8dc08013a87bebbc1afa6c3d09c05b1e4cbad4e0a2384f8b1949050"} Jul 24 13:46:16.291679 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.291636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" event={"ID":"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f","Type":"ContainerStarted","Data":"e2b9ce3c7e17e0d836a6139ca178b16e206bf67583f884739a131f868f753394"} Jul 24 13:46:16.338839 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.338797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zxwf\" (UniqueName: \"kubernetes.io/projected/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-kube-api-access-5zxwf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.339055 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.338915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.339055 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.338979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.339404 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.339380 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.339404 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.339394 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.347399 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.347372 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zxwf\" (UniqueName: \"kubernetes.io/projected/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-kube-api-access-5zxwf\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.497264 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.497231 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:16.581681 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.581644 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7"] Jul 24 13:46:16.585887 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.585861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.594982 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.594939 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7"] Jul 24 13:46:16.641656 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.641624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.641853 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.641663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.641853 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.641689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f268c\" (UniqueName: \"kubernetes.io/projected/08220aaa-4d1f-406b-b09d-df1f4c22db64-kube-api-access-f268c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.647465 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.647441 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6"] Jul 24 13:46:16.649035 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:46:16.649007 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8ebba4_8ff0_446a_b60c_bc223a1239ff.slice/crio-8bbf41120b5959ed0a09cbe717bf871a5a0ee2453e496f113f6c2e0d209d0175 WatchSource:0}: Error finding container 8bbf41120b5959ed0a09cbe717bf871a5a0ee2453e496f113f6c2e0d209d0175: Status 404 returned error can't find the container with id 8bbf41120b5959ed0a09cbe717bf871a5a0ee2453e496f113f6c2e0d209d0175 Jul 24 13:46:16.742229 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.742188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.742229 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.742233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.742511 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.742265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f268c\" (UniqueName: \"kubernetes.io/projected/08220aaa-4d1f-406b-b09d-df1f4c22db64-kube-api-access-f268c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.742703 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.742679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.742817 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.742688 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.751316 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.751286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f268c\" (UniqueName: \"kubernetes.io/projected/08220aaa-4d1f-406b-b09d-df1f4c22db64-kube-api-access-f268c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:16.899387 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:16.899349 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:17.039374 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.039348 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7"] Jul 24 13:46:17.041306 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:46:17.041276 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08220aaa_4d1f_406b_b09d_df1f4c22db64.slice/crio-bac2d5226748d0019d69dff47b5f840399fadbe14050a7834e54d6bef528379f WatchSource:0}: Error finding container bac2d5226748d0019d69dff47b5f840399fadbe14050a7834e54d6bef528379f: Status 404 returned error can't find the container with id bac2d5226748d0019d69dff47b5f840399fadbe14050a7834e54d6bef528379f Jul 24 13:46:17.299003 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.298970 2568 generic.go:356] "Generic (PLEG): container finished" podID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerID="0862db2971683bec161a7e77529f21c847021425628b7ef584e068d65d2c93a4" exitCode=0 Jul 24 13:46:17.299131 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.299029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" event={"ID":"08220aaa-4d1f-406b-b09d-df1f4c22db64","Type":"ContainerDied","Data":"0862db2971683bec161a7e77529f21c847021425628b7ef584e068d65d2c93a4"} Jul 24 13:46:17.299131 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.299089 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" event={"ID":"08220aaa-4d1f-406b-b09d-df1f4c22db64","Type":"ContainerStarted","Data":"bac2d5226748d0019d69dff47b5f840399fadbe14050a7834e54d6bef528379f"} Jul 24 13:46:17.300705 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.300682 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerID="e33477aa07d90fe41118e6546e4cac20ec543c86cdd55286c27df481aa5b2a05" exitCode=0 Jul 24 13:46:17.300828 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.300711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" event={"ID":"1a8ebba4-8ff0-446a-b60c-bc223a1239ff","Type":"ContainerDied","Data":"e33477aa07d90fe41118e6546e4cac20ec543c86cdd55286c27df481aa5b2a05"} Jul 24 13:46:17.300828 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.300776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" event={"ID":"1a8ebba4-8ff0-446a-b60c-bc223a1239ff","Type":"ContainerStarted","Data":"8bbf41120b5959ed0a09cbe717bf871a5a0ee2453e496f113f6c2e0d209d0175"} Jul 24 13:46:17.302605 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.302580 2568 generic.go:356] "Generic (PLEG): container finished" podID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerID="c0930cbd1bcf426630e4a1cb318151b99e88d66f4e2f3953d98c20204da155b8" exitCode=0 Jul 24 13:46:17.302701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:17.302622 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" event={"ID":"4750570c-2073-4de1-a505-7ea27b5d7ff3","Type":"ContainerDied","Data":"c0930cbd1bcf426630e4a1cb318151b99e88d66f4e2f3953d98c20204da155b8"} Jul 24 13:46:18.309643 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.309562 2568 generic.go:356] "Generic (PLEG): container finished" podID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerID="86d40ea8d5557cab007d9a75ef19286760c5eaf67a826704f92bbc91a69e9d90" exitCode=0 Jul 24 13:46:18.310027 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.309658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" event={"ID":"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f","Type":"ContainerDied","Data":"86d40ea8d5557cab007d9a75ef19286760c5eaf67a826704f92bbc91a69e9d90"} Jul 24 13:46:18.311859 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.311836 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" event={"ID":"08220aaa-4d1f-406b-b09d-df1f4c22db64","Type":"ContainerStarted","Data":"efa9a53be21047bb7b21baac5f88427a6e455336b68747f975be60a7197183c7"} Jul 24 13:46:18.313510 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.313475 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerID="daf9d789aaf75ad81f5b5a3c76524cd0461a60be401b2302e9bbc05c62b3e0a0" exitCode=0 Jul 24 13:46:18.313593 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.313529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" event={"ID":"1a8ebba4-8ff0-446a-b60c-bc223a1239ff","Type":"ContainerDied","Data":"daf9d789aaf75ad81f5b5a3c76524cd0461a60be401b2302e9bbc05c62b3e0a0"} Jul 24 13:46:18.315503 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.315479 2568 generic.go:356] "Generic (PLEG): container finished" podID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerID="aa8922de477f1063b5d23180e7b4606a3cf1fea1fc88689000697fc0820c2605" exitCode=0 Jul 24 13:46:18.315607 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:18.315555 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" event={"ID":"4750570c-2073-4de1-a505-7ea27b5d7ff3","Type":"ContainerDied","Data":"aa8922de477f1063b5d23180e7b4606a3cf1fea1fc88689000697fc0820c2605"} Jul 24 13:46:19.322340 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.322307 2568 generic.go:356] "Generic (PLEG): container finished" podID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerID="f6aca242c60542f7e789ef0a6466870ec06d10a1e3f34394128148d9069f2c43" exitCode=0 Jul 24 13:46:19.322822 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.322401 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" event={"ID":"4750570c-2073-4de1-a505-7ea27b5d7ff3","Type":"ContainerDied","Data":"f6aca242c60542f7e789ef0a6466870ec06d10a1e3f34394128148d9069f2c43"} Jul 24 13:46:19.324449 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.324425 2568 generic.go:356] "Generic (PLEG): container finished" podID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerID="c1089fb37fc76a0c979d182cdf272f8819b11ec4447d14a11e65d1790e64d8fa" exitCode=0 Jul 24 13:46:19.324556 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.324503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" event={"ID":"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f","Type":"ContainerDied","Data":"c1089fb37fc76a0c979d182cdf272f8819b11ec4447d14a11e65d1790e64d8fa"} Jul 24 13:46:19.326475 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.326447 2568 generic.go:356] "Generic (PLEG): container finished" podID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerID="efa9a53be21047bb7b21baac5f88427a6e455336b68747f975be60a7197183c7" exitCode=0 Jul 24 13:46:19.326609 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.326521 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" event={"ID":"08220aaa-4d1f-406b-b09d-df1f4c22db64","Type":"ContainerDied","Data":"efa9a53be21047bb7b21baac5f88427a6e455336b68747f975be60a7197183c7"} Jul 24 13:46:19.328808 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.328782 2568 generic.go:356] "Generic (PLEG): container finished" podID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerID="73797653a15a48071393c909d55da302bb9be520058fa45f24ba12647346f15a" exitCode=0 Jul 24 13:46:19.328898 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:19.328845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" event={"ID":"1a8ebba4-8ff0-446a-b60c-bc223a1239ff","Type":"ContainerDied","Data":"73797653a15a48071393c909d55da302bb9be520058fa45f24ba12647346f15a"} Jul 24 13:46:20.336312 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.336278 2568 generic.go:356] "Generic (PLEG): container finished" podID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerID="1ee7ab2c03f1a96600b47909abd44c35f933d9a45475998b6e3549e94853ae9b" exitCode=0 Jul 24 13:46:20.336767 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.336365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" event={"ID":"08220aaa-4d1f-406b-b09d-df1f4c22db64","Type":"ContainerDied","Data":"1ee7ab2c03f1a96600b47909abd44c35f933d9a45475998b6e3549e94853ae9b"} Jul 24 13:46:20.517866 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.517836 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:20.556581 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.556550 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:20.559875 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.559851 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:20.680949 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.680910 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-bundle\") pod \"4750570c-2073-4de1-a505-7ea27b5d7ff3\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " Jul 24 13:46:20.680949 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.680952 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-bundle\") pod \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.680975 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/4750570c-2073-4de1-a505-7ea27b5d7ff3-kube-api-access-8np2d\") pod \"4750570c-2073-4de1-a505-7ea27b5d7ff3\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681001 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-util\") pod \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681022 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-util\") pod \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681047 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-898xs\" (UniqueName: \"kubernetes.io/projected/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-kube-api-access-898xs\") pod \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\" (UID: \"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681071 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-bundle\") pod \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681143 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zxwf\" (UniqueName: \"kubernetes.io/projected/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-kube-api-access-5zxwf\") pod \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\" (UID: \"1a8ebba4-8ff0-446a-b60c-bc223a1239ff\") " Jul 24 13:46:20.681211 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681185 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-util\") pod \"4750570c-2073-4de1-a505-7ea27b5d7ff3\" (UID: \"4750570c-2073-4de1-a505-7ea27b5d7ff3\") " Jul 24 13:46:20.681620 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681525 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-bundle" (OuterVolumeSpecName: "bundle") pod "c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" (UID: "c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:20.682028 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.681996 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-bundle" (OuterVolumeSpecName: "bundle") pod "4750570c-2073-4de1-a505-7ea27b5d7ff3" (UID: "4750570c-2073-4de1-a505-7ea27b5d7ff3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:20.682140 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.682102 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-bundle" (OuterVolumeSpecName: "bundle") pod "1a8ebba4-8ff0-446a-b60c-bc223a1239ff" (UID: "1a8ebba4-8ff0-446a-b60c-bc223a1239ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:20.683740 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.683695 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-kube-api-access-898xs" (OuterVolumeSpecName: "kube-api-access-898xs") pod "c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" (UID: "c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f"). InnerVolumeSpecName "kube-api-access-898xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:46:20.683894 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.683873 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4750570c-2073-4de1-a505-7ea27b5d7ff3-kube-api-access-8np2d" (OuterVolumeSpecName: "kube-api-access-8np2d") pod "4750570c-2073-4de1-a505-7ea27b5d7ff3" (UID: "4750570c-2073-4de1-a505-7ea27b5d7ff3"). InnerVolumeSpecName "kube-api-access-8np2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:46:20.683997 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.683979 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-kube-api-access-5zxwf" (OuterVolumeSpecName: "kube-api-access-5zxwf") pod "1a8ebba4-8ff0-446a-b60c-bc223a1239ff" (UID: "1a8ebba4-8ff0-446a-b60c-bc223a1239ff"). InnerVolumeSpecName "kube-api-access-5zxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:46:20.687208 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.687184 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-util" (OuterVolumeSpecName: "util") pod "4750570c-2073-4de1-a505-7ea27b5d7ff3" (UID: "4750570c-2073-4de1-a505-7ea27b5d7ff3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:20.687425 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.687406 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-util" (OuterVolumeSpecName: "util") pod "1a8ebba4-8ff0-446a-b60c-bc223a1239ff" (UID: "1a8ebba4-8ff0-446a-b60c-bc223a1239ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:20.687640 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.687619 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-util" (OuterVolumeSpecName: "util") pod "c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" (UID: "c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:20.782895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782857 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.782895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782888 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.782895 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782899 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8np2d\" (UniqueName: \"kubernetes.io/projected/4750570c-2073-4de1-a505-7ea27b5d7ff3-kube-api-access-8np2d\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.783139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782908 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.783139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782917 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.783139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782925 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-898xs\" (UniqueName: \"kubernetes.io/projected/c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f-kube-api-access-898xs\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.783139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782933 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.783139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782941 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zxwf\" (UniqueName: \"kubernetes.io/projected/1a8ebba4-8ff0-446a-b60c-bc223a1239ff-kube-api-access-5zxwf\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:20.783139 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:20.782948 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4750570c-2073-4de1-a505-7ea27b5d7ff3-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:21.343931 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.343905 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" Jul 24 13:46:21.344379 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.343933 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph" event={"ID":"4750570c-2073-4de1-a505-7ea27b5d7ff3","Type":"ContainerDied","Data":"5bec695f66490b46d389fba7c4fa833ac0b5c2fde0dc24b85da90eb67e35d29b"} Jul 24 13:46:21.344379 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.343982 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bec695f66490b46d389fba7c4fa833ac0b5c2fde0dc24b85da90eb67e35d29b" Jul 24 13:46:21.345993 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.345971 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" Jul 24 13:46:21.346134 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.345992 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw" event={"ID":"c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f","Type":"ContainerDied","Data":"e2b9ce3c7e17e0d836a6139ca178b16e206bf67583f884739a131f868f753394"} Jul 24 13:46:21.346134 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.346034 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b9ce3c7e17e0d836a6139ca178b16e206bf67583f884739a131f868f753394" Jul 24 13:46:21.348190 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.348164 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" event={"ID":"1a8ebba4-8ff0-446a-b60c-bc223a1239ff","Type":"ContainerDied","Data":"8bbf41120b5959ed0a09cbe717bf871a5a0ee2453e496f113f6c2e0d209d0175"} Jul 24 13:46:21.348293 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.348191 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6" Jul 24 13:46:21.348293 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.348201 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bbf41120b5959ed0a09cbe717bf871a5a0ee2453e496f113f6c2e0d209d0175" Jul 24 13:46:21.480831 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.480795 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:21.591038 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.591004 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-util\") pod \"08220aaa-4d1f-406b-b09d-df1f4c22db64\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " Jul 24 13:46:21.591245 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.591060 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-bundle\") pod \"08220aaa-4d1f-406b-b09d-df1f4c22db64\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " Jul 24 13:46:21.591245 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.591154 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f268c\" (UniqueName: \"kubernetes.io/projected/08220aaa-4d1f-406b-b09d-df1f4c22db64-kube-api-access-f268c\") pod \"08220aaa-4d1f-406b-b09d-df1f4c22db64\" (UID: \"08220aaa-4d1f-406b-b09d-df1f4c22db64\") " Jul 24 13:46:21.591566 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.591542 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-bundle" (OuterVolumeSpecName: "bundle") pod "08220aaa-4d1f-406b-b09d-df1f4c22db64" (UID: "08220aaa-4d1f-406b-b09d-df1f4c22db64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:21.593213 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.593187 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08220aaa-4d1f-406b-b09d-df1f4c22db64-kube-api-access-f268c" (OuterVolumeSpecName: "kube-api-access-f268c") pod "08220aaa-4d1f-406b-b09d-df1f4c22db64" (UID: "08220aaa-4d1f-406b-b09d-df1f4c22db64"). InnerVolumeSpecName "kube-api-access-f268c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:46:21.596899 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.596825 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-util" (OuterVolumeSpecName: "util") pod "08220aaa-4d1f-406b-b09d-df1f4c22db64" (UID: "08220aaa-4d1f-406b-b09d-df1f4c22db64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 13:46:21.692552 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.692511 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:21.692552 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.692549 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f268c\" (UniqueName: \"kubernetes.io/projected/08220aaa-4d1f-406b-b09d-df1f4c22db64-kube-api-access-f268c\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:21.692552 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:21.692561 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08220aaa-4d1f-406b-b09d-df1f4c22db64-util\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:46:22.356099 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:22.356012 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" Jul 24 13:46:22.356099 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:22.356034 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7" event={"ID":"08220aaa-4d1f-406b-b09d-df1f4c22db64","Type":"ContainerDied","Data":"bac2d5226748d0019d69dff47b5f840399fadbe14050a7834e54d6bef528379f"} Jul 24 13:46:22.356099 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:22.356084 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac2d5226748d0019d69dff47b5f840399fadbe14050a7834e54d6bef528379f" Jul 24 13:46:33.360201 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360164 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f6449495-ntc8w"] Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360602 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360621 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360633 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="pull" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360641 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="pull" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360663 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360673 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360681 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="extract" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360689 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="extract" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360702 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="extract" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360709 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="extract" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360740 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360748 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360756 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="pull" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360765 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="pull" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360774 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="extract" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360781 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="extract" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360792 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="util" Jul 24 13:46:33.360809 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360800 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="util" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360823 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="pull" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360831 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="pull" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360842 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="pull" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360850 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="pull" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360860 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="extract" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360868 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="extract" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360966 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4750570c-2073-4de1-a505-7ea27b5d7ff3" containerName="extract" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360978 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="08220aaa-4d1f-406b-b09d-df1f4c22db64" containerName="extract" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360985 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8ebba4-8ff0-446a-b60c-bc223a1239ff" containerName="extract" Jul 24 13:46:33.361709 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.360992 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f" containerName="extract" Jul 24 13:46:33.365531 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.365507 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.383237 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.383201 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f6449495-ntc8w"] Jul 24 13:46:33.399559 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-serving-cert\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.399812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399573 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-oauth-serving-cert\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.399812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-config\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.399812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwgh\" (UniqueName: \"kubernetes.io/projected/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-kube-api-access-9jwgh\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.399812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399716 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-trusted-ca-bundle\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.399812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-service-ca\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.399812 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.399812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-oauth-config\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500318 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-serving-cert\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-oauth-serving-cert\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500377 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-config\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwgh\" (UniqueName: \"kubernetes.io/projected/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-kube-api-access-9jwgh\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-trusted-ca-bundle\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-service-ca\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.500788 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.500549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-oauth-config\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.501705 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.501663 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-service-ca\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.501891 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.501840 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-config\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.504844 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.502011 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-trusted-ca-bundle\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.504844 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.504112 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-serving-cert\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.505203 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.505147 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-console-oauth-config\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.505658 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.505442 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-oauth-serving-cert\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.519617 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.519579 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwgh\" (UniqueName: \"kubernetes.io/projected/f12e7f22-8e04-49d3-bed0-60f9a1f145c5-kube-api-access-9jwgh\") pod \"console-84f6449495-ntc8w\" (UID: \"f12e7f22-8e04-49d3-bed0-60f9a1f145c5\") " pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.675799 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.675766 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:33.819360 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:33.819326 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f6449495-ntc8w"] Jul 24 13:46:33.821462 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:46:33.821436 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12e7f22_8e04_49d3_bed0_60f9a1f145c5.slice/crio-42f92d7f3dce0658655645edeef0e41a00b55a58b57d002abbcc17caecd0d93c WatchSource:0}: Error finding container 42f92d7f3dce0658655645edeef0e41a00b55a58b57d002abbcc17caecd0d93c: Status 404 returned error can't find the container with id 42f92d7f3dce0658655645edeef0e41a00b55a58b57d002abbcc17caecd0d93c Jul 24 13:46:34.423069 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:34.423022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f6449495-ntc8w" event={"ID":"f12e7f22-8e04-49d3-bed0-60f9a1f145c5","Type":"ContainerStarted","Data":"beb01f990ea811e8138f5ec496bf152fc663498c746135ec68110c3b27522ad7"} Jul 24 13:46:34.423446 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:34.423087 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f6449495-ntc8w" event={"ID":"f12e7f22-8e04-49d3-bed0-60f9a1f145c5","Type":"ContainerStarted","Data":"42f92d7f3dce0658655645edeef0e41a00b55a58b57d002abbcc17caecd0d93c"} Jul 24 13:46:34.443069 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:34.443010 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84f6449495-ntc8w" podStartSLOduration=1.442993078 podStartE2EDuration="1.442993078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:46:34.440427403 +0000 UTC m=+491.050907175" watchObservedRunningTime="2026-07-24 13:46:34.442993078 +0000 UTC m=+491.053472823" Jul 24 13:46:43.676690 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:43.676651 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:43.677135 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:43.676757 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:43.681584 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:43.681557 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:44.481196 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:44.481165 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f6449495-ntc8w" Jul 24 13:46:44.533886 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:46:44.533846 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c47b44689-nrffw"] Jul 24 13:47:09.554199 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.554133 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c47b44689-nrffw" podUID="da2119de-dbde-4560-9e8a-7ecaebad0afb" containerName="console" containerID="cri-o://e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a" gracePeriod=15 Jul 24 13:47:09.818349 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.818277 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:47:09.951071 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951023 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-oauth-config\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951112 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-trusted-ca-bundle\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951182 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-oauth-serving-cert\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951224 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-config\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951263 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951252 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r25jd\" (UniqueName: \"kubernetes.io/projected/da2119de-dbde-4560-9e8a-7ecaebad0afb-kube-api-access-r25jd\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951339 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-serving-cert\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951532 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951389 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-service-ca\") pod \"da2119de-dbde-4560-9e8a-7ecaebad0afb\" (UID: \"da2119de-dbde-4560-9e8a-7ecaebad0afb\") " Jul 24 13:47:09.951760 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951704 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-config" (OuterVolumeSpecName: "console-config") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:47:09.951760 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951688 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:47:09.951967 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951840 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-service-ca" (OuterVolumeSpecName: "service-ca") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:47:09.951967 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951869 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-trusted-ca-bundle\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:09.951967 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951885 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:09.951967 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.951837 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 13:47:09.953447 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.953421 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:47:09.953627 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.953599 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 13:47:09.953720 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:09.953660 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2119de-dbde-4560-9e8a-7ecaebad0afb-kube-api-access-r25jd" (OuterVolumeSpecName: "kube-api-access-r25jd") pod "da2119de-dbde-4560-9e8a-7ecaebad0afb" (UID: "da2119de-dbde-4560-9e8a-7ecaebad0afb"). InnerVolumeSpecName "kube-api-access-r25jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 13:47:10.052841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.052805 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-oauth-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:10.052841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.052839 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25jd\" (UniqueName: \"kubernetes.io/projected/da2119de-dbde-4560-9e8a-7ecaebad0afb-kube-api-access-r25jd\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:10.052841 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.052849 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-serving-cert\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:10.053092 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.052858 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da2119de-dbde-4560-9e8a-7ecaebad0afb-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:10.053092 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.052868 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da2119de-dbde-4560-9e8a-7ecaebad0afb-console-oauth-config\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 13:47:10.626477 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.626440 2568 generic.go:356] "Generic (PLEG): container finished" podID="da2119de-dbde-4560-9e8a-7ecaebad0afb" containerID="e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a" exitCode=0 Jul 24 13:47:10.626940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.626489 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c47b44689-nrffw" Jul 24 13:47:10.626940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.626532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47b44689-nrffw" event={"ID":"da2119de-dbde-4560-9e8a-7ecaebad0afb","Type":"ContainerDied","Data":"e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a"} Jul 24 13:47:10.626940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.626586 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c47b44689-nrffw" event={"ID":"da2119de-dbde-4560-9e8a-7ecaebad0afb","Type":"ContainerDied","Data":"8de98ecd92709589fd69d3c4151326e7867682a5b1c7ff361e2a43fc3e0aeb6d"} Jul 24 13:47:10.626940 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.626608 2568 scope.go:117] "RemoveContainer" containerID="e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a" Jul 24 13:47:10.636217 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.636196 2568 scope.go:117] "RemoveContainer" containerID="e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a" Jul 24 13:47:10.636503 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:47:10.636485 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a\": container with ID starting with e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a not found: ID does not exist" containerID="e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a" Jul 24 13:47:10.636559 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.636513 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a"} err="failed to get container status \"e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a\": rpc error: code = NotFound desc = could not find container \"e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a\": container with ID starting with e96d16a63f015eead6f1661bdf6878535f5a958791aef76c7911d2aa1aaaa58a not found: ID does not exist" Jul 24 13:47:10.648476 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.648437 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c47b44689-nrffw"] Jul 24 13:47:10.652358 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:10.652329 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c47b44689-nrffw"] Jul 24 13:47:11.860355 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:47:11.860322 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2119de-dbde-4560-9e8a-7ecaebad0afb" path="/var/lib/kubelet/pods/da2119de-dbde-4560-9e8a-7ecaebad0afb/volumes" Jul 24 13:49:12.009122 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.009080 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6"] Jul 24 13:49:12.009630 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.009469 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2119de-dbde-4560-9e8a-7ecaebad0afb" containerName="console" Jul 24 13:49:12.009630 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.009482 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2119de-dbde-4560-9e8a-7ecaebad0afb" containerName="console" Jul 24 13:49:12.009630 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.009559 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2119de-dbde-4560-9e8a-7ecaebad0afb" containerName="console" Jul 24 13:49:12.012735 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.012711 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.015216 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.015187 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wtgtp\"" Jul 24 13:49:12.015216 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.015202 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 13:49:12.015412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.015228 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 13:49:12.015412 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.015189 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 13:49:12.024108 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.024076 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6"] Jul 24 13:49:12.080576 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.080541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0559ef72-57d2-44a9-bd4f-2db513596604-maas-api-tls\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.080776 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.080626 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0559ef72-57d2-44a9-bd4f-2db513596604-openshift-service-ca\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.080776 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.080694 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9gsw\" (UniqueName: \"kubernetes.io/projected/0559ef72-57d2-44a9-bd4f-2db513596604-kube-api-access-f9gsw\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.181779 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.181704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0559ef72-57d2-44a9-bd4f-2db513596604-maas-api-tls\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.181779 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.181782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0559ef72-57d2-44a9-bd4f-2db513596604-openshift-service-ca\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.182064 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.181820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9gsw\" (UniqueName: \"kubernetes.io/projected/0559ef72-57d2-44a9-bd4f-2db513596604-kube-api-access-f9gsw\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.182064 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:49:12.181851 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 24 13:49:12.182064 ip-10-0-138-248 kubenswrapper[2568]: E0724 13:49:12.181914 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0559ef72-57d2-44a9-bd4f-2db513596604-maas-api-tls podName:0559ef72-57d2-44a9-bd4f-2db513596604 nodeName:}" failed. No retries permitted until 2026-07-24 13:49:12.681897473 +0000 UTC m=+649.292377201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0559ef72-57d2-44a9-bd4f-2db513596604-maas-api-tls") pod "maas-api-f95b6f759-5k9t6" (UID: "0559ef72-57d2-44a9-bd4f-2db513596604") : secret "maas-api-serving-cert" not found Jul 24 13:49:12.182573 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.182547 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0559ef72-57d2-44a9-bd4f-2db513596604-openshift-service-ca\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.190315 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.190285 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9gsw\" (UniqueName: \"kubernetes.io/projected/0559ef72-57d2-44a9-bd4f-2db513596604-kube-api-access-f9gsw\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.686273 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.686231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0559ef72-57d2-44a9-bd4f-2db513596604-maas-api-tls\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.688851 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.688815 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0559ef72-57d2-44a9-bd4f-2db513596604-maas-api-tls\") pod \"maas-api-f95b6f759-5k9t6\" (UID: \"0559ef72-57d2-44a9-bd4f-2db513596604\") " pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:12.924973 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:12.924940 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:13.068971 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:13.068941 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6"] Jul 24 13:49:13.071494 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:49:13.071466 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0559ef72_57d2_44a9_bd4f_2db513596604.slice/crio-dc6d8f631388167dd8a3776119620ce1eadf6a0e0be2f727afe0f9ed01072d04 WatchSource:0}: Error finding container dc6d8f631388167dd8a3776119620ce1eadf6a0e0be2f727afe0f9ed01072d04: Status 404 returned error can't find the container with id dc6d8f631388167dd8a3776119620ce1eadf6a0e0be2f727afe0f9ed01072d04 Jul 24 13:49:13.073283 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:13.073252 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 13:49:13.289654 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:13.289567 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" event={"ID":"0559ef72-57d2-44a9-bd4f-2db513596604","Type":"ContainerStarted","Data":"dc6d8f631388167dd8a3776119620ce1eadf6a0e0be2f727afe0f9ed01072d04"} Jul 24 13:49:16.309297 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:16.309255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" event={"ID":"0559ef72-57d2-44a9-bd4f-2db513596604","Type":"ContainerStarted","Data":"c4b9940932b0b2992dc8bdb4b942d702930c71c576628aeffd222a12ed0d0de5"} Jul 24 13:49:16.309741 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:16.309402 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:16.326156 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:16.326098 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" podStartSLOduration=3.044947878 podStartE2EDuration="5.326077517s" totalImagesPullingTime="2.281129639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:49:16.325760987 +0000 UTC m=+652.936240733" watchObservedRunningTime="2026-07-24 13:49:16.326077517 +0000 UTC m=+652.936557261" Jul 24 13:49:22.321776 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:22.321745 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-f95b6f759-5k9t6" Jul 24 13:49:39.844378 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.844338 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6"] Jul 24 13:49:39.852265 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.852237 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:39.856224 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.856199 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9qk2t\"" Jul 24 13:49:39.857028 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.857008 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 13:49:39.857151 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.857129 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 13:49:39.857222 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.857167 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 13:49:39.864352 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.864325 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6"] Jul 24 13:49:39.938593 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.938558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:39.938804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.938600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:39.938804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.938632 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a99fcd-f54d-404c-8f74-adf733edf234-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:39.938804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.938692 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:39.938804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.938771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:39.938804 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:39.938800 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkplw\" (UniqueName: \"kubernetes.io/projected/20a99fcd-f54d-404c-8f74-adf733edf234-kube-api-access-zkplw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039444 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a99fcd-f54d-404c-8f74-adf733edf234-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039444 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039440 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkplw\" (UniqueName: \"kubernetes.io/projected/20a99fcd-f54d-404c-8f74-adf733edf234-kube-api-access-zkplw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039701 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.039960 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039922 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.040016 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.039960 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.040098 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.040078 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.041736 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.041701 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20a99fcd-f54d-404c-8f74-adf733edf234-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.042020 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.042000 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20a99fcd-f54d-404c-8f74-adf733edf234-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.048217 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.048185 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkplw\" (UniqueName: \"kubernetes.io/projected/20a99fcd-f54d-404c-8f74-adf733edf234-kube-api-access-zkplw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-92sv6\" (UID: \"20a99fcd-f54d-404c-8f74-adf733edf234\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.164894 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.164830 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:40.304493 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.304456 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6"] Jul 24 13:49:40.307875 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:49:40.307848 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a99fcd_f54d_404c_8f74_adf733edf234.slice/crio-e81208dfd4c7ad099f7d42107af9ec341e94d79eac314cee50afb0fbeab563bd WatchSource:0}: Error finding container e81208dfd4c7ad099f7d42107af9ec341e94d79eac314cee50afb0fbeab563bd: Status 404 returned error can't find the container with id e81208dfd4c7ad099f7d42107af9ec341e94d79eac314cee50afb0fbeab563bd Jul 24 13:49:40.438904 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:40.438806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" event={"ID":"20a99fcd-f54d-404c-8f74-adf733edf234","Type":"ContainerStarted","Data":"e81208dfd4c7ad099f7d42107af9ec341e94d79eac314cee50afb0fbeab563bd"} Jul 24 13:49:41.446333 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:41.446298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" event={"ID":"20a99fcd-f54d-404c-8f74-adf733edf234","Type":"ContainerStarted","Data":"feced0c3fb15cccf39e3c89155fcc80724a81e8fa8454bd8f7036cad56919861"} Jul 24 13:49:41.464478 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:41.464419 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" podStartSLOduration=1.455471969 podStartE2EDuration="2.464401764s" totalImagesPullingTime="1.008929795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:49:41.463082588 +0000 UTC m=+678.073562334" watchObservedRunningTime="2026-07-24 13:49:41.464401764 +0000 UTC m=+678.074881511" Jul 24 13:49:50.165684 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:50.165643 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:50.166165 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:50.165765 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:50.178927 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:50.178893 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:49:50.507321 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:49:50.507239 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-92sv6" Jul 24 13:50:17.018989 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.018889 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm"] Jul 24 13:50:17.023739 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.023704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.026285 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.026255 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 24 13:50:17.032455 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.032427 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm"] Jul 24 13:50:17.088253 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.088213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66324d1-03e8-4363-ad3b-58bb9e309235-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.088253 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.088258 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.088466 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.088291 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dsmd\" (UniqueName: \"kubernetes.io/projected/b66324d1-03e8-4363-ad3b-58bb9e309235-kube-api-access-9dsmd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.088466 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.088317 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.088466 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.088386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.088466 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.088425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.189571 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.189527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.189798 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.189585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dsmd\" (UniqueName: \"kubernetes.io/projected/b66324d1-03e8-4363-ad3b-58bb9e309235-kube-api-access-9dsmd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.189798 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.189621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.189798 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.189672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.189798 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.189699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.189798 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.189790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66324d1-03e8-4363-ad3b-58bb9e309235-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.190078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.190040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.190078 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.190056 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.190173 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.190151 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.192042 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.192008 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b66324d1-03e8-4363-ad3b-58bb9e309235-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.192268 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.192250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b66324d1-03e8-4363-ad3b-58bb9e309235-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.200101 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.200075 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dsmd\" (UniqueName: \"kubernetes.io/projected/b66324d1-03e8-4363-ad3b-58bb9e309235-kube-api-access-9dsmd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-l2lvm\" (UID: \"b66324d1-03e8-4363-ad3b-58bb9e309235\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.335509 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.335411 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:17.481511 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.481478 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm"] Jul 24 13:50:17.482714 ip-10-0-138-248 kubenswrapper[2568]: W0724 13:50:17.482688 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66324d1_03e8_4363_ad3b_58bb9e309235.slice/crio-935a8ddeac843bbaa80a30e88acb6f814000d76f4d13d9e789b43542dc9ca549 WatchSource:0}: Error finding container 935a8ddeac843bbaa80a30e88acb6f814000d76f4d13d9e789b43542dc9ca549: Status 404 returned error can't find the container with id 935a8ddeac843bbaa80a30e88acb6f814000d76f4d13d9e789b43542dc9ca549 Jul 24 13:50:17.642358 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:17.642317 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" event={"ID":"b66324d1-03e8-4363-ad3b-58bb9e309235","Type":"ContainerStarted","Data":"935a8ddeac843bbaa80a30e88acb6f814000d76f4d13d9e789b43542dc9ca549"} Jul 24 13:50:18.649194 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:18.649157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" event={"ID":"b66324d1-03e8-4363-ad3b-58bb9e309235","Type":"ContainerStarted","Data":"53bc236c72897d3d75f5493439ce74deb0873fb615736f684c1d616c10d11373"} Jul 24 13:50:18.672178 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:18.672122 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" podStartSLOduration=2.473273326 podStartE2EDuration="2.672104038s" totalImagesPullingTime="198.830712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 13:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 13:50:18.668779874 +0000 UTC m=+715.279259621" watchObservedRunningTime="2026-07-24 13:50:18.672104038 +0000 UTC m=+715.282583790" Jul 24 13:50:27.335554 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:27.335516 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:27.336019 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:27.335590 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:27.348117 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:27.348089 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 13:50:27.711474 ip-10-0-138-248 kubenswrapper[2568]: I0724 13:50:27.711441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-l2lvm" Jul 24 14:07:19.592664 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.592619 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l"] Jul 24 14:07:19.596059 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.596039 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:19.598885 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.598859 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:07:19.603901 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.603744 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l"] Jul 24 14:07:19.651574 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.651523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l\" (UID: \"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:19.752828 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.752782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l\" (UID: \"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:19.753436 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.753415 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l\" (UID: \"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:19.908339 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:19.908289 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:20.259936 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:20.259908 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l"] Jul 24 14:07:20.262496 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:07:20.262469 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbd6ac8_f562_4f9f_8ac7_3833b355eef8.slice/crio-f93ec07f220cec757a9e702a9046fb25d5207c8db3e8e2c14a0f5bb153145cbf WatchSource:0}: Error finding container f93ec07f220cec757a9e702a9046fb25d5207c8db3e8e2c14a0f5bb153145cbf: Status 404 returned error can't find the container with id f93ec07f220cec757a9e702a9046fb25d5207c8db3e8e2c14a0f5bb153145cbf Jul 24 14:07:20.264391 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:20.264373 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:07:21.265386 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:21.265342 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" event={"ID":"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8","Type":"ContainerStarted","Data":"f93ec07f220cec757a9e702a9046fb25d5207c8db3e8e2c14a0f5bb153145cbf"} Jul 24 14:07:23.279685 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:23.279645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" event={"ID":"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8","Type":"ContainerStarted","Data":"7f840a633d17269e85c575e2aa83500547ca5cd94dfe2ade013ced5333545c63"} Jul 24 14:07:23.297345 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:23.297295 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" podStartSLOduration=2.012240659 podStartE2EDuration="4.297277164s" totalImagesPullingTime="2.285036505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:07:23.296023091 +0000 UTC m=+1739.906502837" watchObservedRunningTime="2026-07-24 14:07:23.297277164 +0000 UTC m=+1739.907756909" Jul 24 14:07:24.286580 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:24.286543 2568 generic.go:356] "Generic (PLEG): container finished" podID="cbbd6ac8-f562-4f9f-8ac7-3833b355eef8" containerID="7f840a633d17269e85c575e2aa83500547ca5cd94dfe2ade013ced5333545c63" exitCode=0 Jul 24 14:07:24.287017 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:24.286626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" event={"ID":"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8","Type":"ContainerDied","Data":"7f840a633d17269e85c575e2aa83500547ca5cd94dfe2ade013ced5333545c63"} Jul 24 14:07:25.426903 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:25.426867 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:25.519636 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:25.519597 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8-maas-api-service-ca\") pod \"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8\" (UID: \"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8\") " Jul 24 14:07:25.519995 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:25.519971 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbbd6ac8-f562-4f9f-8ac7-3833b355eef8" (UID: "cbbd6ac8-f562-4f9f-8ac7-3833b355eef8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:07:25.620720 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:25.620684 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:07:26.299412 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:26.299380 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" Jul 24 14:07:26.299596 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:26.299427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l" event={"ID":"cbbd6ac8-f562-4f9f-8ac7-3833b355eef8","Type":"ContainerDied","Data":"f93ec07f220cec757a9e702a9046fb25d5207c8db3e8e2c14a0f5bb153145cbf"} Jul 24 14:07:26.299596 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:26.299473 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93ec07f220cec757a9e702a9046fb25d5207c8db3e8e2c14a0f5bb153145cbf" Jul 24 14:07:34.674668 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:34.674630 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l"] Jul 24 14:07:34.678719 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:34.678690 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35e2fd78-crc9l"] Jul 24 14:07:35.861014 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:07:35.860977 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbd6ac8-f562-4f9f-8ac7-3833b355eef8" path="/var/lib/kubelet/pods/cbbd6ac8-f562-4f9f-8ac7-3833b355eef8/volumes" Jul 24 14:08:03.355612 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.355571 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6"] Jul 24 14:08:03.356073 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.355970 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbd6ac8-f562-4f9f-8ac7-3833b355eef8" containerName="revoke-keys" Jul 24 14:08:03.356073 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.355981 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbd6ac8-f562-4f9f-8ac7-3833b355eef8" containerName="revoke-keys" Jul 24 14:08:03.356073 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.356055 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbd6ac8-f562-4f9f-8ac7-3833b355eef8" containerName="revoke-keys" Jul 24 14:08:03.364165 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.364138 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:03.366508 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.366476 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6"] Jul 24 14:08:03.367005 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.366981 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:08:03.485940 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.485901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42af87d3-5351-4a3a-a8b5-dc61973b221c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6\" (UID: \"42af87d3-5351-4a3a-a8b5-dc61973b221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:03.586856 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.586820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42af87d3-5351-4a3a-a8b5-dc61973b221c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6\" (UID: \"42af87d3-5351-4a3a-a8b5-dc61973b221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:03.587470 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.587450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42af87d3-5351-4a3a-a8b5-dc61973b221c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6\" (UID: \"42af87d3-5351-4a3a-a8b5-dc61973b221c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:03.676612 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.676577 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:03.816569 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:03.816541 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6"] Jul 24 14:08:03.818920 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:08:03.818891 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42af87d3_5351_4a3a_a8b5_dc61973b221c.slice/crio-80aaf5d35244d8fe550e9cce71ab38f20b0e1ea2bc5b6719d84bf58ed5319c6a WatchSource:0}: Error finding container 80aaf5d35244d8fe550e9cce71ab38f20b0e1ea2bc5b6719d84bf58ed5319c6a: Status 404 returned error can't find the container with id 80aaf5d35244d8fe550e9cce71ab38f20b0e1ea2bc5b6719d84bf58ed5319c6a Jul 24 14:08:04.507878 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:04.507845 2568 generic.go:356] "Generic (PLEG): container finished" podID="42af87d3-5351-4a3a-a8b5-dc61973b221c" containerID="4a13fcc58673933b93ef03136ec37f8c746b448c58c6a12a5d358a34fa786dec" exitCode=0 Jul 24 14:08:04.508251 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:04.507890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" event={"ID":"42af87d3-5351-4a3a-a8b5-dc61973b221c","Type":"ContainerDied","Data":"4a13fcc58673933b93ef03136ec37f8c746b448c58c6a12a5d358a34fa786dec"} Jul 24 14:08:04.508251 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:04.507963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" event={"ID":"42af87d3-5351-4a3a-a8b5-dc61973b221c","Type":"ContainerStarted","Data":"80aaf5d35244d8fe550e9cce71ab38f20b0e1ea2bc5b6719d84bf58ed5319c6a"} Jul 24 14:08:05.653942 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:05.653915 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:05.706534 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:05.706497 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42af87d3-5351-4a3a-a8b5-dc61973b221c-maas-api-service-ca\") pod \"42af87d3-5351-4a3a-a8b5-dc61973b221c\" (UID: \"42af87d3-5351-4a3a-a8b5-dc61973b221c\") " Jul 24 14:08:05.706946 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:05.706920 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42af87d3-5351-4a3a-a8b5-dc61973b221c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42af87d3-5351-4a3a-a8b5-dc61973b221c" (UID: "42af87d3-5351-4a3a-a8b5-dc61973b221c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:05.808225 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:05.808129 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42af87d3-5351-4a3a-a8b5-dc61973b221c-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:08:06.520978 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:06.520947 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" Jul 24 14:08:06.520978 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:06.520960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6" event={"ID":"42af87d3-5351-4a3a-a8b5-dc61973b221c","Type":"ContainerDied","Data":"80aaf5d35244d8fe550e9cce71ab38f20b0e1ea2bc5b6719d84bf58ed5319c6a"} Jul 24 14:08:06.521217 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:06.521006 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80aaf5d35244d8fe550e9cce71ab38f20b0e1ea2bc5b6719d84bf58ed5319c6a" Jul 24 14:08:18.440345 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:18.440285 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6"] Jul 24 14:08:18.442604 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:18.442579 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-83602dfb-pxmj6"] Jul 24 14:08:19.859822 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:19.859783 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42af87d3-5351-4a3a-a8b5-dc61973b221c" path="/var/lib/kubelet/pods/42af87d3-5351-4a3a-a8b5-dc61973b221c/volumes" Jul 24 14:08:22.287000 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.286963 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh"] Jul 24 14:08:22.287492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.287474 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42af87d3-5351-4a3a-a8b5-dc61973b221c" containerName="revoke-keys" Jul 24 14:08:22.287551 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.287495 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="42af87d3-5351-4a3a-a8b5-dc61973b221c" containerName="revoke-keys" Jul 24 14:08:22.287588 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.287582 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="42af87d3-5351-4a3a-a8b5-dc61973b221c" containerName="revoke-keys" Jul 24 14:08:22.290699 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.290680 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.293500 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.293476 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-388084cb\"" Jul 24 14:08:22.293657 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.293608 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-388084cb-dockercfg-qhcnc\"" Jul 24 14:08:22.302127 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.302098 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh"] Jul 24 14:08:22.473840 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.473797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv69\" (UniqueName: \"kubernetes.io/projected/a2670ba7-8679-4a5e-986d-d249103f5ce8-kube-api-access-twv69\") pod \"payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.474035 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.473933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2670ba7-8679-4a5e-986d-d249103f5ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.575527 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.575429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2670ba7-8679-4a5e-986d-d249103f5ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.575697 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.575593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twv69\" (UniqueName: \"kubernetes.io/projected/a2670ba7-8679-4a5e-986d-d249103f5ce8-kube-api-access-twv69\") pod \"payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.576244 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.576220 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2670ba7-8679-4a5e-986d-d249103f5ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.586373 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.586341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv69\" (UniqueName: \"kubernetes.io/projected/a2670ba7-8679-4a5e-986d-d249103f5ce8-kube-api-access-twv69\") pod \"payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.603194 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.603158 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:22.750082 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:22.750048 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh"] Jul 24 14:08:22.751519 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:08:22.751489 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2670ba7_8679_4a5e_986d_d249103f5ce8.slice/crio-c3a3de333a86d41ebe35f08d8d37c8ffd7787f13918776256242059e1921d82f WatchSource:0}: Error finding container c3a3de333a86d41ebe35f08d8d37c8ffd7787f13918776256242059e1921d82f: Status 404 returned error can't find the container with id c3a3de333a86d41ebe35f08d8d37c8ffd7787f13918776256242059e1921d82f Jul 24 14:08:23.626154 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:23.626075 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" event={"ID":"a2670ba7-8679-4a5e-986d-d249103f5ce8","Type":"ContainerStarted","Data":"c3a3de333a86d41ebe35f08d8d37c8ffd7787f13918776256242059e1921d82f"} Jul 24 14:08:24.633716 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:24.633682 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" event={"ID":"a2670ba7-8679-4a5e-986d-d249103f5ce8","Type":"ContainerStarted","Data":"1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524"} Jul 24 14:08:24.634140 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:24.633766 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:24.653587 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:24.653528 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" podStartSLOduration=1.096358134 podStartE2EDuration="2.653511582s" totalImagesPullingTime="1.557153448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:08:24.651389501 +0000 UTC m=+1801.261869250" watchObservedRunningTime="2026-07-24 14:08:24.653511582 +0000 UTC m=+1801.263991327" Jul 24 14:08:35.641314 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:35.641270 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" podUID="a2670ba7-8679-4a5e-986d-d249103f5ce8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.51:9004: connect: connection refused" Jul 24 14:08:45.640806 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:45.640772 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:57.918818 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:57.918777 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh"] Jul 24 14:08:57.919229 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:57.919040 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" podUID="a2670ba7-8679-4a5e-986d-d249103f5ce8" containerName="payload-pre-processing" containerID="cri-o://1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524" gracePeriod=30 Jul 24 14:08:58.181621 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.181548 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:58.310391 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.310360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2670ba7-8679-4a5e-986d-d249103f5ce8-plugins-config-volume\") pod \"a2670ba7-8679-4a5e-986d-d249103f5ce8\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " Jul 24 14:08:58.310391 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.310394 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twv69\" (UniqueName: \"kubernetes.io/projected/a2670ba7-8679-4a5e-986d-d249103f5ce8-kube-api-access-twv69\") pod \"a2670ba7-8679-4a5e-986d-d249103f5ce8\" (UID: \"a2670ba7-8679-4a5e-986d-d249103f5ce8\") " Jul 24 14:08:58.310859 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.310832 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2670ba7-8679-4a5e-986d-d249103f5ce8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2670ba7-8679-4a5e-986d-d249103f5ce8" (UID: "a2670ba7-8679-4a5e-986d-d249103f5ce8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:08:58.312665 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.312637 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2670ba7-8679-4a5e-986d-d249103f5ce8-kube-api-access-twv69" (OuterVolumeSpecName: "kube-api-access-twv69") pod "a2670ba7-8679-4a5e-986d-d249103f5ce8" (UID: "a2670ba7-8679-4a5e-986d-d249103f5ce8"). InnerVolumeSpecName "kube-api-access-twv69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:08:58.411460 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.411422 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2670ba7-8679-4a5e-986d-d249103f5ce8-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.411460 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.411455 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twv69\" (UniqueName: \"kubernetes.io/projected/a2670ba7-8679-4a5e-986d-d249103f5ce8-kube-api-access-twv69\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:08:58.827896 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.827864 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2670ba7-8679-4a5e-986d-d249103f5ce8" containerID="1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524" exitCode=0 Jul 24 14:08:58.827896 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.827901 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" event={"ID":"a2670ba7-8679-4a5e-986d-d249103f5ce8","Type":"ContainerDied","Data":"1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524"} Jul 24 14:08:58.828110 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.827906 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" Jul 24 14:08:58.828110 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.827949 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh" event={"ID":"a2670ba7-8679-4a5e-986d-d249103f5ce8","Type":"ContainerDied","Data":"c3a3de333a86d41ebe35f08d8d37c8ffd7787f13918776256242059e1921d82f"} Jul 24 14:08:58.828110 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.827971 2568 scope.go:117] "RemoveContainer" containerID="1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524" Jul 24 14:08:58.837810 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.837791 2568 scope.go:117] "RemoveContainer" containerID="1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524" Jul 24 14:08:58.838093 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:08:58.838072 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524\": container with ID starting with 1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524 not found: ID does not exist" containerID="1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524" Jul 24 14:08:58.838140 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.838098 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524"} err="failed to get container status \"1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524\": rpc error: code = NotFound desc = could not find container \"1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524\": container with ID starting with 1c651a8a276824630457cd9a957e6b78538fa5bcd1e0307695f5eacefcb23524 not found: ID does not exist" Jul 24 14:08:58.851337 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.851304 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh"] Jul 24 14:08:58.854917 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:58.854889 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-388084cb-599d9cc844-mn5xh"] Jul 24 14:08:59.859600 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:08:59.859565 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2670ba7-8679-4a5e-986d-d249103f5ce8" path="/var/lib/kubelet/pods/a2670ba7-8679-4a5e-986d-d249103f5ce8/volumes" Jul 24 14:09:07.599739 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.599699 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd"] Jul 24 14:09:07.600241 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.600138 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2670ba7-8679-4a5e-986d-d249103f5ce8" containerName="payload-pre-processing" Jul 24 14:09:07.600241 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.600151 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2670ba7-8679-4a5e-986d-d249103f5ce8" containerName="payload-pre-processing" Jul 24 14:09:07.600241 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.600220 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2670ba7-8679-4a5e-986d-d249103f5ce8" containerName="payload-pre-processing" Jul 24 14:09:07.604956 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.604930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.607869 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.607843 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a487c2a7\"" Jul 24 14:09:07.613186 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.612934 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd"] Jul 24 14:09:07.699703 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.699659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm2v\" (UniqueName: \"kubernetes.io/projected/47eac81d-8891-4cf8-ba73-8f6479729904-kube-api-access-zjm2v\") pod \"payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.699931 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.699901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eac81d-8891-4cf8-ba73-8f6479729904-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.801245 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.801202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eac81d-8891-4cf8-ba73-8f6479729904-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.801439 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.801267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm2v\" (UniqueName: \"kubernetes.io/projected/47eac81d-8891-4cf8-ba73-8f6479729904-kube-api-access-zjm2v\") pod \"payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.801991 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.801970 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eac81d-8891-4cf8-ba73-8f6479729904-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.809689 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.809657 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm2v\" (UniqueName: \"kubernetes.io/projected/47eac81d-8891-4cf8-ba73-8f6479729904-kube-api-access-zjm2v\") pod \"payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:07.917571 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:07.917529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:08.076329 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:08.076299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd"] Jul 24 14:09:08.078649 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:09:08.078621 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47eac81d_8891_4cf8_ba73_8f6479729904.slice/crio-fdc6a1a25fe3527fc858cac929f5baa17daf4a97673d75f78b77337a7ee093a9 WatchSource:0}: Error finding container fdc6a1a25fe3527fc858cac929f5baa17daf4a97673d75f78b77337a7ee093a9: Status 404 returned error can't find the container with id fdc6a1a25fe3527fc858cac929f5baa17daf4a97673d75f78b77337a7ee093a9 Jul 24 14:09:08.895361 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:08.895323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" event={"ID":"47eac81d-8891-4cf8-ba73-8f6479729904","Type":"ContainerStarted","Data":"58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a"} Jul 24 14:09:08.895816 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:08.895392 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:08.895816 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:08.895413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" event={"ID":"47eac81d-8891-4cf8-ba73-8f6479729904","Type":"ContainerStarted","Data":"fdc6a1a25fe3527fc858cac929f5baa17daf4a97673d75f78b77337a7ee093a9"} Jul 24 14:09:08.917501 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:08.917443 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" podStartSLOduration=1.917427408 podStartE2EDuration="1.917427408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:09:08.914119073 +0000 UTC m=+1845.524598821" watchObservedRunningTime="2026-07-24 14:09:08.917427408 +0000 UTC m=+1845.527907154" Jul 24 14:09:19.902901 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:19.902867 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:42.193651 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.193605 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd"] Jul 24 14:09:42.194168 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.193880 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" podUID="47eac81d-8891-4cf8-ba73-8f6479729904" containerName="payload-pre-processing" containerID="cri-o://58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a" gracePeriod=30 Jul 24 14:09:42.446965 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.446898 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:42.528397 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.528360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eac81d-8891-4cf8-ba73-8f6479729904-plugins-config-volume\") pod \"47eac81d-8891-4cf8-ba73-8f6479729904\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " Jul 24 14:09:42.528569 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.528408 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjm2v\" (UniqueName: \"kubernetes.io/projected/47eac81d-8891-4cf8-ba73-8f6479729904-kube-api-access-zjm2v\") pod \"47eac81d-8891-4cf8-ba73-8f6479729904\" (UID: \"47eac81d-8891-4cf8-ba73-8f6479729904\") " Jul 24 14:09:42.528803 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.528780 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47eac81d-8891-4cf8-ba73-8f6479729904-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47eac81d-8891-4cf8-ba73-8f6479729904" (UID: "47eac81d-8891-4cf8-ba73-8f6479729904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:09:42.530507 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.530485 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eac81d-8891-4cf8-ba73-8f6479729904-kube-api-access-zjm2v" (OuterVolumeSpecName: "kube-api-access-zjm2v") pod "47eac81d-8891-4cf8-ba73-8f6479729904" (UID: "47eac81d-8891-4cf8-ba73-8f6479729904"). InnerVolumeSpecName "kube-api-access-zjm2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:09:42.629942 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.629901 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47eac81d-8891-4cf8-ba73-8f6479729904-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:09:42.629942 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:42.629930 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm2v\" (UniqueName: \"kubernetes.io/projected/47eac81d-8891-4cf8-ba73-8f6479729904-kube-api-access-zjm2v\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:09:43.088305 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.088266 2568 generic.go:356] "Generic (PLEG): container finished" podID="47eac81d-8891-4cf8-ba73-8f6479729904" containerID="58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a" exitCode=0 Jul 24 14:09:43.088484 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.088325 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" Jul 24 14:09:43.088484 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.088348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" event={"ID":"47eac81d-8891-4cf8-ba73-8f6479729904","Type":"ContainerDied","Data":"58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a"} Jul 24 14:09:43.088484 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.088399 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd" event={"ID":"47eac81d-8891-4cf8-ba73-8f6479729904","Type":"ContainerDied","Data":"fdc6a1a25fe3527fc858cac929f5baa17daf4a97673d75f78b77337a7ee093a9"} Jul 24 14:09:43.088484 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.088421 2568 scope.go:117] "RemoveContainer" containerID="58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a" Jul 24 14:09:43.097913 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.097892 2568 scope.go:117] "RemoveContainer" containerID="58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a" Jul 24 14:09:43.098226 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:09:43.098207 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a\": container with ID starting with 58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a not found: ID does not exist" containerID="58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a" Jul 24 14:09:43.098286 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.098236 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a"} err="failed to get container status \"58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a\": rpc error: code = NotFound desc = could not find container \"58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a\": container with ID starting with 58c92b63574fa5378267257b798663263106d0d6bd6375cc18d6344e4008e94a not found: ID does not exist" Jul 24 14:09:43.110283 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.110255 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd"] Jul 24 14:09:43.114243 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.114216 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a487c2a7-64447f7f5b-5b5nd"] Jul 24 14:09:43.860158 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:09:43.860115 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47eac81d-8891-4cf8-ba73-8f6479729904" path="/var/lib/kubelet/pods/47eac81d-8891-4cf8-ba73-8f6479729904/volumes" Jul 24 14:13:20.960093 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.960059 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727"] Jul 24 14:13:20.960593 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.960429 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47eac81d-8891-4cf8-ba73-8f6479729904" containerName="payload-pre-processing" Jul 24 14:13:20.960593 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.960440 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="47eac81d-8891-4cf8-ba73-8f6479729904" containerName="payload-pre-processing" Jul 24 14:13:20.960593 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.960516 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="47eac81d-8891-4cf8-ba73-8f6479729904" containerName="payload-pre-processing" Jul 24 14:13:20.965480 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.965449 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:20.970954 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.970922 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:13:20.972811 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:20.972778 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727"] Jul 24 14:13:21.101291 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:21.101251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6c669d-14da-43df-8972-21b9ac8460f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-54be8ac5-7f727\" (UID: \"ae6c669d-14da-43df-8972-21b9ac8460f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:21.202717 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:21.202674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6c669d-14da-43df-8972-21b9ac8460f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-54be8ac5-7f727\" (UID: \"ae6c669d-14da-43df-8972-21b9ac8460f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:21.203349 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:21.203327 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6c669d-14da-43df-8972-21b9ac8460f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-54be8ac5-7f727\" (UID: \"ae6c669d-14da-43df-8972-21b9ac8460f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:21.279896 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:21.279815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:21.425380 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:21.425350 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727"] Jul 24 14:13:21.427431 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:13:21.427404 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6c669d_14da_43df_8972_21b9ac8460f4.slice/crio-cb7bd4374b7b538e373d5cf2443aeac2de4bc65618ac50a9e339981329f48ac2 WatchSource:0}: Error finding container cb7bd4374b7b538e373d5cf2443aeac2de4bc65618ac50a9e339981329f48ac2: Status 404 returned error can't find the container with id cb7bd4374b7b538e373d5cf2443aeac2de4bc65618ac50a9e339981329f48ac2 Jul 24 14:13:22.323851 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:22.323761 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae6c669d-14da-43df-8972-21b9ac8460f4" containerID="18a3b4c48de166cd5f2b6a9f14278110b6f633fdb126e0048f15ea8e4125e5f6" exitCode=0 Jul 24 14:13:22.324216 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:22.323859 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" event={"ID":"ae6c669d-14da-43df-8972-21b9ac8460f4","Type":"ContainerDied","Data":"18a3b4c48de166cd5f2b6a9f14278110b6f633fdb126e0048f15ea8e4125e5f6"} Jul 24 14:13:22.324216 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:22.323907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" event={"ID":"ae6c669d-14da-43df-8972-21b9ac8460f4","Type":"ContainerStarted","Data":"cb7bd4374b7b538e373d5cf2443aeac2de4bc65618ac50a9e339981329f48ac2"} Jul 24 14:13:23.463941 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:23.463915 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:23.525261 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:23.525226 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6c669d-14da-43df-8972-21b9ac8460f4-maas-api-service-ca\") pod \"ae6c669d-14da-43df-8972-21b9ac8460f4\" (UID: \"ae6c669d-14da-43df-8972-21b9ac8460f4\") " Jul 24 14:13:23.525600 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:23.525572 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6c669d-14da-43df-8972-21b9ac8460f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae6c669d-14da-43df-8972-21b9ac8460f4" (UID: "ae6c669d-14da-43df-8972-21b9ac8460f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:13:23.626096 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:23.626069 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae6c669d-14da-43df-8972-21b9ac8460f4-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:13:24.035906 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:24.035822 2568 scope.go:117] "RemoveContainer" containerID="7f840a633d17269e85c575e2aa83500547ca5cd94dfe2ade013ced5333545c63" Jul 24 14:13:24.337269 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:24.337188 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" Jul 24 14:13:24.337269 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:24.337208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727" event={"ID":"ae6c669d-14da-43df-8972-21b9ac8460f4","Type":"ContainerDied","Data":"cb7bd4374b7b538e373d5cf2443aeac2de4bc65618ac50a9e339981329f48ac2"} Jul 24 14:13:24.337269 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:24.337255 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb7bd4374b7b538e373d5cf2443aeac2de4bc65618ac50a9e339981329f48ac2" Jul 24 14:13:41.153030 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:41.152988 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727"] Jul 24 14:13:41.155444 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:41.155418 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-54be8ac5-7f727"] Jul 24 14:13:41.860131 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:13:41.860090 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6c669d-14da-43df-8972-21b9ac8460f4" path="/var/lib/kubelet/pods/ae6c669d-14da-43df-8972-21b9ac8460f4/volumes" Jul 24 14:14:24.057297 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:14:24.057259 2568 scope.go:117] "RemoveContainer" containerID="4a13fcc58673933b93ef03136ec37f8c746b448c58c6a12a5d358a34fa786dec" Jul 24 14:15:49.321746 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.321635 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2"] Jul 24 14:15:49.322197 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.322035 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6c669d-14da-43df-8972-21b9ac8460f4" containerName="revoke-keys" Jul 24 14:15:49.322197 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.322047 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6c669d-14da-43df-8972-21b9ac8460f4" containerName="revoke-keys" Jul 24 14:15:49.322197 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.322122 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6c669d-14da-43df-8972-21b9ac8460f4" containerName="revoke-keys" Jul 24 14:15:49.325298 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.325280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.328059 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.328024 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-406bb9\"" Jul 24 14:15:49.328298 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.328280 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-406bb9-dockercfg-hwb4z\"" Jul 24 14:15:49.337234 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.337203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2"] Jul 24 14:15:49.349183 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.349143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ff264c-735f-4eaf-81f3-f272675921df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.349374 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.349205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp765\" (UniqueName: \"kubernetes.io/projected/80ff264c-735f-4eaf-81f3-f272675921df-kube-api-access-dp765\") pod \"payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.449887 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.449835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ff264c-735f-4eaf-81f3-f272675921df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.450091 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.449916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp765\" (UniqueName: \"kubernetes.io/projected/80ff264c-735f-4eaf-81f3-f272675921df-kube-api-access-dp765\") pod \"payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.450545 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.450524 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ff264c-735f-4eaf-81f3-f272675921df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.459048 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.459011 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp765\" (UniqueName: \"kubernetes.io/projected/80ff264c-735f-4eaf-81f3-f272675921df-kube-api-access-dp765\") pod \"payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.636675 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.636640 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:49.992133 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:49.992016 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2"] Jul 24 14:15:49.994544 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:15:49.994514 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ff264c_735f_4eaf_81f3_f272675921df.slice/crio-777431079ae366e1c7e17988ddb3de0541005783a54bcd32a7aa401a0c3fb6b6 WatchSource:0}: Error finding container 777431079ae366e1c7e17988ddb3de0541005783a54bcd32a7aa401a0c3fb6b6: Status 404 returned error can't find the container with id 777431079ae366e1c7e17988ddb3de0541005783a54bcd32a7aa401a0c3fb6b6 Jul 24 14:15:50.152383 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:50.152348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" event={"ID":"80ff264c-735f-4eaf-81f3-f272675921df","Type":"ContainerStarted","Data":"436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610"} Jul 24 14:15:50.152602 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:50.152409 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" event={"ID":"80ff264c-735f-4eaf-81f3-f272675921df","Type":"ContainerStarted","Data":"777431079ae366e1c7e17988ddb3de0541005783a54bcd32a7aa401a0c3fb6b6"} Jul 24 14:15:50.152602 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:50.152465 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:15:50.172711 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:15:50.172653 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" podStartSLOduration=1.172633808 podStartE2EDuration="1.172633808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:15:50.170967142 +0000 UTC m=+2246.781446889" watchObservedRunningTime="2026-07-24 14:15:50.172633808 +0000 UTC m=+2246.783113555" Jul 24 14:16:01.160888 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:16:01.160853 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:17:14.069635 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.069597 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk"] Jul 24 14:17:14.072971 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.072951 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:14.075479 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.075454 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:17:14.080889 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.080856 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk"] Jul 24 14:17:14.157719 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.157677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk\" (UID: \"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:14.258819 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.258775 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk\" (UID: \"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:14.259456 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.259433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk\" (UID: \"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:14.387680 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.387636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:14.538054 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.538027 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk"] Jul 24 14:17:14.539929 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:17:14.539901 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8eab5b_da2d_4ac4_bf29_891cfd8076c1.slice/crio-78d95fa4c962919eded8ee11e98e2b6851f928003e7c1b64c6659fe24daaa5ee WatchSource:0}: Error finding container 78d95fa4c962919eded8ee11e98e2b6851f928003e7c1b64c6659fe24daaa5ee: Status 404 returned error can't find the container with id 78d95fa4c962919eded8ee11e98e2b6851f928003e7c1b64c6659fe24daaa5ee Jul 24 14:17:14.624927 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:14.624887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" event={"ID":"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1","Type":"ContainerStarted","Data":"78d95fa4c962919eded8ee11e98e2b6851f928003e7c1b64c6659fe24daaa5ee"} Jul 24 14:17:15.631566 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:15.631529 2568 generic.go:356] "Generic (PLEG): container finished" podID="fa8eab5b-da2d-4ac4-bf29-891cfd8076c1" containerID="81f24a3d0c79446a2cf0c19ca3eced60b9c51625ac2cf8a3fdbf91f74c3040f4" exitCode=0 Jul 24 14:17:15.631983 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:15.631625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" event={"ID":"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1","Type":"ContainerDied","Data":"81f24a3d0c79446a2cf0c19ca3eced60b9c51625ac2cf8a3fdbf91f74c3040f4"} Jul 24 14:17:16.784218 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:16.784179 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:16.882568 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:16.882525 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1-maas-api-service-ca\") pod \"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1\" (UID: \"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1\") " Jul 24 14:17:16.882964 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:16.882940 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa8eab5b-da2d-4ac4-bf29-891cfd8076c1" (UID: "fa8eab5b-da2d-4ac4-bf29-891cfd8076c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:16.984042 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:16.983947 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:17:17.644037 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:17.644005 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" Jul 24 14:17:17.644220 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:17.644040 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk" event={"ID":"fa8eab5b-da2d-4ac4-bf29-891cfd8076c1","Type":"ContainerDied","Data":"78d95fa4c962919eded8ee11e98e2b6851f928003e7c1b64c6659fe24daaa5ee"} Jul 24 14:17:17.644220 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:17.644083 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d95fa4c962919eded8ee11e98e2b6851f928003e7c1b64c6659fe24daaa5ee" Jul 24 14:17:29.145785 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:29.145712 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk"] Jul 24 14:17:29.148048 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:29.148018 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6304ef-99wzk"] Jul 24 14:17:29.860256 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:29.860220 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8eab5b-da2d-4ac4-bf29-891cfd8076c1" path="/var/lib/kubelet/pods/fa8eab5b-da2d-4ac4-bf29-891cfd8076c1/volumes" Jul 24 14:17:36.331571 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.331466 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86"] Jul 24 14:17:36.333946 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.332090 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8eab5b-da2d-4ac4-bf29-891cfd8076c1" containerName="revoke-keys" Jul 24 14:17:36.333946 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.332109 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8eab5b-da2d-4ac4-bf29-891cfd8076c1" containerName="revoke-keys" Jul 24 14:17:36.333946 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.332191 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8eab5b-da2d-4ac4-bf29-891cfd8076c1" containerName="revoke-keys" Jul 24 14:17:36.335107 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.335086 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:36.337919 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.337893 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:17:36.346647 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.346614 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86"] Jul 24 14:17:36.479162 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.479119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f556d-9d6b-4328-8dc5-75896dd97ca2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86\" (UID: \"7c5f556d-9d6b-4328-8dc5-75896dd97ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:36.580625 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.580578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f556d-9d6b-4328-8dc5-75896dd97ca2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86\" (UID: \"7c5f556d-9d6b-4328-8dc5-75896dd97ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:36.581303 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.581280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f556d-9d6b-4328-8dc5-75896dd97ca2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86\" (UID: \"7c5f556d-9d6b-4328-8dc5-75896dd97ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:36.647885 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.647852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:36.799160 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:36.798993 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86"] Jul 24 14:17:36.802252 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:17:36.802222 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5f556d_9d6b_4328_8dc5_75896dd97ca2.slice/crio-54974a8122bcbd8a9ddbc7c89648e2e157719d04aebb955653aad9e748a5cdc4 WatchSource:0}: Error finding container 54974a8122bcbd8a9ddbc7c89648e2e157719d04aebb955653aad9e748a5cdc4: Status 404 returned error can't find the container with id 54974a8122bcbd8a9ddbc7c89648e2e157719d04aebb955653aad9e748a5cdc4 Jul 24 14:17:37.762715 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:37.762606 2568 generic.go:356] "Generic (PLEG): container finished" podID="7c5f556d-9d6b-4328-8dc5-75896dd97ca2" containerID="449f7582bc19235a79e2510798ba5c4f9b0802188cba392573e6f80f1b7f67b7" exitCode=0 Jul 24 14:17:37.762715 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:37.762700 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" event={"ID":"7c5f556d-9d6b-4328-8dc5-75896dd97ca2","Type":"ContainerDied","Data":"449f7582bc19235a79e2510798ba5c4f9b0802188cba392573e6f80f1b7f67b7"} Jul 24 14:17:37.763144 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:37.762779 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" event={"ID":"7c5f556d-9d6b-4328-8dc5-75896dd97ca2","Type":"ContainerStarted","Data":"54974a8122bcbd8a9ddbc7c89648e2e157719d04aebb955653aad9e748a5cdc4"} Jul 24 14:17:38.909235 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:38.909209 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:39.001917 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:39.001869 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f556d-9d6b-4328-8dc5-75896dd97ca2-maas-api-service-ca\") pod \"7c5f556d-9d6b-4328-8dc5-75896dd97ca2\" (UID: \"7c5f556d-9d6b-4328-8dc5-75896dd97ca2\") " Jul 24 14:17:39.002346 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:39.002305 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c5f556d-9d6b-4328-8dc5-75896dd97ca2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c5f556d-9d6b-4328-8dc5-75896dd97ca2" (UID: "7c5f556d-9d6b-4328-8dc5-75896dd97ca2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:39.103789 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:39.103670 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c5f556d-9d6b-4328-8dc5-75896dd97ca2-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:17:39.775089 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:39.775058 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" Jul 24 14:17:39.775268 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:39.775061 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86" event={"ID":"7c5f556d-9d6b-4328-8dc5-75896dd97ca2","Type":"ContainerDied","Data":"54974a8122bcbd8a9ddbc7c89648e2e157719d04aebb955653aad9e748a5cdc4"} Jul 24 14:17:39.775268 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:39.775183 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54974a8122bcbd8a9ddbc7c89648e2e157719d04aebb955653aad9e748a5cdc4" Jul 24 14:17:46.921883 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:46.921842 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2"] Jul 24 14:17:46.922753 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:46.922653 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" podUID="80ff264c-735f-4eaf-81f3-f272675921df" containerName="payload-pre-processing" containerID="cri-o://436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610" gracePeriod=30 Jul 24 14:17:47.175619 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.175554 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:17:47.282044 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.282010 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ff264c-735f-4eaf-81f3-f272675921df-plugins-config-volume\") pod \"80ff264c-735f-4eaf-81f3-f272675921df\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " Jul 24 14:17:47.282225 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.282072 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp765\" (UniqueName: \"kubernetes.io/projected/80ff264c-735f-4eaf-81f3-f272675921df-kube-api-access-dp765\") pod \"80ff264c-735f-4eaf-81f3-f272675921df\" (UID: \"80ff264c-735f-4eaf-81f3-f272675921df\") " Jul 24 14:17:47.282441 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.282409 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ff264c-735f-4eaf-81f3-f272675921df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80ff264c-735f-4eaf-81f3-f272675921df" (UID: "80ff264c-735f-4eaf-81f3-f272675921df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:17:47.284360 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.284334 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ff264c-735f-4eaf-81f3-f272675921df-kube-api-access-dp765" (OuterVolumeSpecName: "kube-api-access-dp765") pod "80ff264c-735f-4eaf-81f3-f272675921df" (UID: "80ff264c-735f-4eaf-81f3-f272675921df"). InnerVolumeSpecName "kube-api-access-dp765". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:17:47.383665 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.383621 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ff264c-735f-4eaf-81f3-f272675921df-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:17:47.383665 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.383655 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp765\" (UniqueName: \"kubernetes.io/projected/80ff264c-735f-4eaf-81f3-f272675921df-kube-api-access-dp765\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:17:47.827412 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.827373 2568 generic.go:356] "Generic (PLEG): container finished" podID="80ff264c-735f-4eaf-81f3-f272675921df" containerID="436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610" exitCode=0 Jul 24 14:17:47.827597 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.827427 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" Jul 24 14:17:47.827597 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.827467 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" event={"ID":"80ff264c-735f-4eaf-81f3-f272675921df","Type":"ContainerDied","Data":"436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610"} Jul 24 14:17:47.827597 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.827524 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2" event={"ID":"80ff264c-735f-4eaf-81f3-f272675921df","Type":"ContainerDied","Data":"777431079ae366e1c7e17988ddb3de0541005783a54bcd32a7aa401a0c3fb6b6"} Jul 24 14:17:47.827597 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.827553 2568 scope.go:117] "RemoveContainer" containerID="436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610" Jul 24 14:17:47.837576 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.837551 2568 scope.go:117] "RemoveContainer" containerID="436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610" Jul 24 14:17:47.837918 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:17:47.837896 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610\": container with ID starting with 436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610 not found: ID does not exist" containerID="436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610" Jul 24 14:17:47.837998 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.837932 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610"} err="failed to get container status \"436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610\": rpc error: code = NotFound desc = could not find container \"436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610\": container with ID starting with 436119d1bdb8693532b282c171d765f401a325ff723f0203489e504f98e6f610 not found: ID does not exist" Jul 24 14:17:47.851293 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.851260 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2"] Jul 24 14:17:47.860264 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:47.860234 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-406bb9-c95665bfb-rx4b2"] Jul 24 14:17:49.860999 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:49.860966 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ff264c-735f-4eaf-81f3-f272675921df" path="/var/lib/kubelet/pods/80ff264c-735f-4eaf-81f3-f272675921df/volumes" Jul 24 14:17:51.405298 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:51.405257 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86"] Jul 24 14:17:51.409482 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:51.409442 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-406bb9-tsk86"] Jul 24 14:17:51.859311 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:17:51.859226 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c5f556d-9d6b-4328-8dc5-75896dd97ca2" path="/var/lib/kubelet/pods/7c5f556d-9d6b-4328-8dc5-75896dd97ca2/volumes" Jul 24 14:18:11.223772 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.223736 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69"] Jul 24 14:18:11.224238 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.224153 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c5f556d-9d6b-4328-8dc5-75896dd97ca2" containerName="revoke-keys" Jul 24 14:18:11.224238 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.224166 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c5f556d-9d6b-4328-8dc5-75896dd97ca2" containerName="revoke-keys" Jul 24 14:18:11.224238 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.224184 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ff264c-735f-4eaf-81f3-f272675921df" containerName="payload-pre-processing" Jul 24 14:18:11.224238 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.224199 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ff264c-735f-4eaf-81f3-f272675921df" containerName="payload-pre-processing" Jul 24 14:18:11.224373 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.224264 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ff264c-735f-4eaf-81f3-f272675921df" containerName="payload-pre-processing" Jul 24 14:18:11.224373 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.224273 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c5f556d-9d6b-4328-8dc5-75896dd97ca2" containerName="revoke-keys" Jul 24 14:18:11.227411 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.227392 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.229996 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.229973 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-517d91\"" Jul 24 14:18:11.230180 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.230167 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-517d91-dockercfg-94drh\"" Jul 24 14:18:11.239880 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.239851 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69"] Jul 24 14:18:11.297339 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.297302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6376fe-94cf-4bcf-9519-27a73a960ed6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.297540 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.297445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qdw\" (UniqueName: \"kubernetes.io/projected/4e6376fe-94cf-4bcf-9519-27a73a960ed6-kube-api-access-m7qdw\") pod \"payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.399043 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.399000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6376fe-94cf-4bcf-9519-27a73a960ed6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.399241 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.399140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qdw\" (UniqueName: \"kubernetes.io/projected/4e6376fe-94cf-4bcf-9519-27a73a960ed6-kube-api-access-m7qdw\") pod \"payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.399754 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.399710 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6376fe-94cf-4bcf-9519-27a73a960ed6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.407346 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.407325 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qdw\" (UniqueName: \"kubernetes.io/projected/4e6376fe-94cf-4bcf-9519-27a73a960ed6-kube-api-access-m7qdw\") pod \"payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.538396 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.538300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.690233 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.690203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69"] Jul 24 14:18:11.691716 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:18:11.691691 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6376fe_94cf_4bcf_9519_27a73a960ed6.slice/crio-61919c70390a63811bcd77d32bd636058375bbed129e33b7211c5ec75f4a0652 WatchSource:0}: Error finding container 61919c70390a63811bcd77d32bd636058375bbed129e33b7211c5ec75f4a0652: Status 404 returned error can't find the container with id 61919c70390a63811bcd77d32bd636058375bbed129e33b7211c5ec75f4a0652 Jul 24 14:18:11.963594 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.963555 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" event={"ID":"4e6376fe-94cf-4bcf-9519-27a73a960ed6","Type":"ContainerStarted","Data":"519b6deef658f1c2cc77aba405918abce2b30f618235f6632bcbe7e7746a102a"} Jul 24 14:18:11.963809 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.963610 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" event={"ID":"4e6376fe-94cf-4bcf-9519-27a73a960ed6","Type":"ContainerStarted","Data":"61919c70390a63811bcd77d32bd636058375bbed129e33b7211c5ec75f4a0652"} Jul 24 14:18:11.963809 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.963677 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:11.984529 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:11.984472 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" podStartSLOduration=0.984455686 podStartE2EDuration="984.455686ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:11.981689689 +0000 UTC m=+2388.592169435" watchObservedRunningTime="2026-07-24 14:18:11.984455686 +0000 UTC m=+2388.594935423" Jul 24 14:18:22.971235 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:22.971201 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:18:23.085259 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.085220 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl"] Jul 24 14:18:23.088962 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.088944 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:23.091569 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.091552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:18:23.098019 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.097991 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl"] Jul 24 14:18:23.219083 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.219042 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74345b13-a9f4-4d64-9372-283fd5b2a552-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e05489-nhmcl\" (UID: \"74345b13-a9f4-4d64-9372-283fd5b2a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:23.320085 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.319978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74345b13-a9f4-4d64-9372-283fd5b2a552-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e05489-nhmcl\" (UID: \"74345b13-a9f4-4d64-9372-283fd5b2a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:23.320641 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.320619 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74345b13-a9f4-4d64-9372-283fd5b2a552-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e05489-nhmcl\" (UID: \"74345b13-a9f4-4d64-9372-283fd5b2a552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:23.401877 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.401834 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:23.546263 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:23.546238 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl"] Jul 24 14:18:23.549019 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:18:23.548972 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74345b13_a9f4_4d64_9372_283fd5b2a552.slice/crio-65fa2a089892d63ccde616e5933ffae6f3e8e4028803d224e76d0e3a47e24124 WatchSource:0}: Error finding container 65fa2a089892d63ccde616e5933ffae6f3e8e4028803d224e76d0e3a47e24124: Status 404 returned error can't find the container with id 65fa2a089892d63ccde616e5933ffae6f3e8e4028803d224e76d0e3a47e24124 Jul 24 14:18:24.040431 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:24.040392 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" event={"ID":"74345b13-a9f4-4d64-9372-283fd5b2a552","Type":"ContainerStarted","Data":"91587129afaea54c5e6fa060553a276eebdde486d04bb55d168f45fb31307492"} Jul 24 14:18:24.040926 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:24.040494 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" event={"ID":"74345b13-a9f4-4d64-9372-283fd5b2a552","Type":"ContainerStarted","Data":"65fa2a089892d63ccde616e5933ffae6f3e8e4028803d224e76d0e3a47e24124"} Jul 24 14:18:24.071671 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:24.071619 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" podStartSLOduration=1.0716027399999999 podStartE2EDuration="1.07160274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:24.069822549 +0000 UTC m=+2400.680302295" watchObservedRunningTime="2026-07-24 14:18:24.07160274 +0000 UTC m=+2400.682082488" Jul 24 14:18:25.048392 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:25.048354 2568 generic.go:356] "Generic (PLEG): container finished" podID="74345b13-a9f4-4d64-9372-283fd5b2a552" containerID="91587129afaea54c5e6fa060553a276eebdde486d04bb55d168f45fb31307492" exitCode=0 Jul 24 14:18:25.048820 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:25.048439 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" event={"ID":"74345b13-a9f4-4d64-9372-283fd5b2a552","Type":"ContainerDied","Data":"91587129afaea54c5e6fa060553a276eebdde486d04bb55d168f45fb31307492"} Jul 24 14:18:26.196031 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:26.196002 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:26.352262 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:26.352162 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74345b13-a9f4-4d64-9372-283fd5b2a552-maas-api-service-ca\") pod \"74345b13-a9f4-4d64-9372-283fd5b2a552\" (UID: \"74345b13-a9f4-4d64-9372-283fd5b2a552\") " Jul 24 14:18:26.352599 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:26.352575 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74345b13-a9f4-4d64-9372-283fd5b2a552-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74345b13-a9f4-4d64-9372-283fd5b2a552" (UID: "74345b13-a9f4-4d64-9372-283fd5b2a552"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:26.453131 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:26.453090 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74345b13-a9f4-4d64-9372-283fd5b2a552-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:18:27.060939 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:27.060902 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" Jul 24 14:18:27.060939 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:27.060929 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl" event={"ID":"74345b13-a9f4-4d64-9372-283fd5b2a552","Type":"ContainerDied","Data":"65fa2a089892d63ccde616e5933ffae6f3e8e4028803d224e76d0e3a47e24124"} Jul 24 14:18:27.061154 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:27.061000 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65fa2a089892d63ccde616e5933ffae6f3e8e4028803d224e76d0e3a47e24124" Jul 24 14:18:43.181228 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:43.181186 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl"] Jul 24 14:18:43.183451 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:43.183427 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e05489-nhmcl"] Jul 24 14:18:43.867811 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:43.867779 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74345b13-a9f4-4d64-9372-283fd5b2a552" path="/var/lib/kubelet/pods/74345b13-a9f4-4d64-9372-283fd5b2a552/volumes" Jul 24 14:18:50.217800 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.217706 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr"] Jul 24 14:18:50.218178 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.218108 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74345b13-a9f4-4d64-9372-283fd5b2a552" containerName="revoke-keys" Jul 24 14:18:50.218178 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.218120 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="74345b13-a9f4-4d64-9372-283fd5b2a552" containerName="revoke-keys" Jul 24 14:18:50.218257 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.218201 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="74345b13-a9f4-4d64-9372-283fd5b2a552" containerName="revoke-keys" Jul 24 14:18:50.221365 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.221347 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:50.223959 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.223937 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:18:50.228918 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.228889 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr"] Jul 24 14:18:50.256528 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.256480 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517d91-4ccjr\" (UID: \"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:50.358031 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.357986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517d91-4ccjr\" (UID: \"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:50.358630 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.358606 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517d91-4ccjr\" (UID: \"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:50.533280 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.533203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:50.679465 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:50.679435 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr"] Jul 24 14:18:50.681436 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:18:50.681404 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b14f9f_63ab_42db_ae3c_cbcfad8f80b8.slice/crio-5af5c6891decfc3c17a5857a96d1f09e73ed0a89101391c65a417b1c48500c6c WatchSource:0}: Error finding container 5af5c6891decfc3c17a5857a96d1f09e73ed0a89101391c65a417b1c48500c6c: Status 404 returned error can't find the container with id 5af5c6891decfc3c17a5857a96d1f09e73ed0a89101391c65a417b1c48500c6c Jul 24 14:18:51.200460 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:51.200358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" event={"ID":"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8","Type":"ContainerStarted","Data":"61305ee14ff227e27c26041401bf3b1e945c5d40444547870be22a12c0fd25c6"} Jul 24 14:18:51.200460 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:51.200420 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" event={"ID":"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8","Type":"ContainerStarted","Data":"5af5c6891decfc3c17a5857a96d1f09e73ed0a89101391c65a417b1c48500c6c"} Jul 24 14:18:51.217956 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:51.217901 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" podStartSLOduration=1.217883701 podStartE2EDuration="1.217883701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:18:51.216083076 +0000 UTC m=+2427.826562823" watchObservedRunningTime="2026-07-24 14:18:51.217883701 +0000 UTC m=+2427.828363447" Jul 24 14:18:52.208160 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:52.208122 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8" containerID="61305ee14ff227e27c26041401bf3b1e945c5d40444547870be22a12c0fd25c6" exitCode=0 Jul 24 14:18:52.208352 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:52.208170 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" event={"ID":"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8","Type":"ContainerDied","Data":"61305ee14ff227e27c26041401bf3b1e945c5d40444547870be22a12c0fd25c6"} Jul 24 14:18:53.352629 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:53.352604 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:53.387457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:53.387424 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8-maas-api-service-ca\") pod \"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8\" (UID: \"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8\") " Jul 24 14:18:53.387805 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:53.387784 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8" (UID: "c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:18:53.488575 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:53.488480 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:18:54.220784 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:54.220678 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" Jul 24 14:18:54.220784 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:54.220750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr" event={"ID":"c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8","Type":"ContainerDied","Data":"5af5c6891decfc3c17a5857a96d1f09e73ed0a89101391c65a417b1c48500c6c"} Jul 24 14:18:54.221003 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:18:54.220798 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5af5c6891decfc3c17a5857a96d1f09e73ed0a89101391c65a417b1c48500c6c" Jul 24 14:19:06.157915 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.157870 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69"] Jul 24 14:19:06.158559 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.158107 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" podUID="4e6376fe-94cf-4bcf-9519-27a73a960ed6" containerName="payload-pre-processing" containerID="cri-o://519b6deef658f1c2cc77aba405918abce2b30f618235f6632bcbe7e7746a102a" gracePeriod=30 Jul 24 14:19:06.290797 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.290756 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" event={"ID":"4e6376fe-94cf-4bcf-9519-27a73a960ed6","Type":"ContainerDied","Data":"519b6deef658f1c2cc77aba405918abce2b30f618235f6632bcbe7e7746a102a"} Jul 24 14:19:06.290953 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.290713 2568 generic.go:356] "Generic (PLEG): container finished" podID="4e6376fe-94cf-4bcf-9519-27a73a960ed6" containerID="519b6deef658f1c2cc77aba405918abce2b30f618235f6632bcbe7e7746a102a" exitCode=0 Jul 24 14:19:06.427799 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.427775 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:19:06.511493 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.511452 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6376fe-94cf-4bcf-9519-27a73a960ed6-plugins-config-volume\") pod \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " Jul 24 14:19:06.511671 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.511587 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7qdw\" (UniqueName: \"kubernetes.io/projected/4e6376fe-94cf-4bcf-9519-27a73a960ed6-kube-api-access-m7qdw\") pod \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\" (UID: \"4e6376fe-94cf-4bcf-9519-27a73a960ed6\") " Jul 24 14:19:06.511909 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.511885 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e6376fe-94cf-4bcf-9519-27a73a960ed6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e6376fe-94cf-4bcf-9519-27a73a960ed6" (UID: "4e6376fe-94cf-4bcf-9519-27a73a960ed6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:19:06.513648 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.513623 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6376fe-94cf-4bcf-9519-27a73a960ed6-kube-api-access-m7qdw" (OuterVolumeSpecName: "kube-api-access-m7qdw") pod "4e6376fe-94cf-4bcf-9519-27a73a960ed6" (UID: "4e6376fe-94cf-4bcf-9519-27a73a960ed6"). InnerVolumeSpecName "kube-api-access-m7qdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:19:06.612627 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.612592 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7qdw\" (UniqueName: \"kubernetes.io/projected/4e6376fe-94cf-4bcf-9519-27a73a960ed6-kube-api-access-m7qdw\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:19:06.612627 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:06.612624 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e6376fe-94cf-4bcf-9519-27a73a960ed6-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:19:07.297984 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:07.297952 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" Jul 24 14:19:07.298408 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:07.297952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69" event={"ID":"4e6376fe-94cf-4bcf-9519-27a73a960ed6","Type":"ContainerDied","Data":"61919c70390a63811bcd77d32bd636058375bbed129e33b7211c5ec75f4a0652"} Jul 24 14:19:07.298408 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:07.298118 2568 scope.go:117] "RemoveContainer" containerID="519b6deef658f1c2cc77aba405918abce2b30f618235f6632bcbe7e7746a102a" Jul 24 14:19:07.323073 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:07.323033 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69"] Jul 24 14:19:07.325328 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:07.325305 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-517d91-6fbff954d-9pk69"] Jul 24 14:19:07.863332 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:07.863300 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6376fe-94cf-4bcf-9519-27a73a960ed6" path="/var/lib/kubelet/pods/4e6376fe-94cf-4bcf-9519-27a73a960ed6/volumes" Jul 24 14:19:10.297769 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:10.297705 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr"] Jul 24 14:19:10.300224 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:10.300195 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517d91-4ccjr"] Jul 24 14:19:11.860165 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:11.860133 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8" path="/var/lib/kubelet/pods/c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8/volumes" Jul 24 14:19:21.609212 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.609154 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg"] Jul 24 14:19:21.610100 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.610073 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8" containerName="revoke-keys" Jul 24 14:19:21.610100 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.610101 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8" containerName="revoke-keys" Jul 24 14:19:21.610261 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.610120 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6376fe-94cf-4bcf-9519-27a73a960ed6" containerName="payload-pre-processing" Jul 24 14:19:21.610261 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.610128 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6376fe-94cf-4bcf-9519-27a73a960ed6" containerName="payload-pre-processing" Jul 24 14:19:21.610261 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.610245 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6376fe-94cf-4bcf-9519-27a73a960ed6" containerName="payload-pre-processing" Jul 24 14:19:21.610261 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.610257 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6b14f9f-63ab-42db-ae3c-cbcfad8f80b8" containerName="revoke-keys" Jul 24 14:19:21.614836 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.614809 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.617683 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.617659 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a26cce\"" Jul 24 14:19:21.617837 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.617659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a26cce-dockercfg-9kspk\"" Jul 24 14:19:21.625132 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.625106 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg"] Jul 24 14:19:21.660443 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.660403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46afc43a-1618-4d36-bc69-3540964b5783-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.660636 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.660536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmvh\" (UniqueName: \"kubernetes.io/projected/46afc43a-1618-4d36-bc69-3540964b5783-kube-api-access-zvmvh\") pod \"payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.761661 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.761624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmvh\" (UniqueName: \"kubernetes.io/projected/46afc43a-1618-4d36-bc69-3540964b5783-kube-api-access-zvmvh\") pod \"payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.761661 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.761666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46afc43a-1618-4d36-bc69-3540964b5783-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.762439 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.762418 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46afc43a-1618-4d36-bc69-3540964b5783-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.770237 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.770207 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmvh\" (UniqueName: \"kubernetes.io/projected/46afc43a-1618-4d36-bc69-3540964b5783-kube-api-access-zvmvh\") pod \"payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:21.927805 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:21.927766 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:22.069469 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:22.069437 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg"] Jul 24 14:19:22.072693 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:19:22.072662 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46afc43a_1618_4d36_bc69_3540964b5783.slice/crio-aa38772dd9fdf294a981d652101f58708386b523ef7ea07e3013db93258e29e5 WatchSource:0}: Error finding container aa38772dd9fdf294a981d652101f58708386b523ef7ea07e3013db93258e29e5: Status 404 returned error can't find the container with id aa38772dd9fdf294a981d652101f58708386b523ef7ea07e3013db93258e29e5 Jul 24 14:19:22.387127 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:22.387092 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" event={"ID":"46afc43a-1618-4d36-bc69-3540964b5783","Type":"ContainerStarted","Data":"5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b"} Jul 24 14:19:22.387308 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:22.387177 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" event={"ID":"46afc43a-1618-4d36-bc69-3540964b5783","Type":"ContainerStarted","Data":"aa38772dd9fdf294a981d652101f58708386b523ef7ea07e3013db93258e29e5"} Jul 24 14:19:22.387308 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:22.387205 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:19:22.405464 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:22.405406 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" podStartSLOduration=1.405391352 podStartE2EDuration="1.405391352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:22.403758799 +0000 UTC m=+2459.014238546" watchObservedRunningTime="2026-07-24 14:19:22.405391352 +0000 UTC m=+2459.015871098" Jul 24 14:19:24.123049 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:24.123012 2568 scope.go:117] "RemoveContainer" containerID="18a3b4c48de166cd5f2b6a9f14278110b6f633fdb126e0048f15ea8e4125e5f6" Jul 24 14:19:29.765049 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.765014 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds"] Jul 24 14:19:29.768166 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.768149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:29.770960 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.770934 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-cb9aa5\"" Jul 24 14:19:29.771080 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.771056 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-cb9aa5-dockercfg-tnnzh\"" Jul 24 14:19:29.778360 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.778328 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds"] Jul 24 14:19:29.831963 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.831927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lzr\" (UniqueName: \"kubernetes.io/projected/11a139cb-afad-4ec8-b567-541387bbe0c1-kube-api-access-q5lzr\") pod \"payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:29.832144 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.831992 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a139cb-afad-4ec8-b567-541387bbe0c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:29.932834 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.932798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lzr\" (UniqueName: \"kubernetes.io/projected/11a139cb-afad-4ec8-b567-541387bbe0c1-kube-api-access-q5lzr\") pod \"payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:29.933020 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.932871 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a139cb-afad-4ec8-b567-541387bbe0c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:29.933657 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.933636 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a139cb-afad-4ec8-b567-541387bbe0c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:29.941801 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:29.941768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lzr\" (UniqueName: \"kubernetes.io/projected/11a139cb-afad-4ec8-b567-541387bbe0c1-kube-api-access-q5lzr\") pod \"payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:30.081095 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:30.080999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:30.225581 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:30.225548 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds"] Jul 24 14:19:30.227562 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:19:30.227534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a139cb_afad_4ec8_b567_541387bbe0c1.slice/crio-2c416e36acbb3c03e43fc53d66c440b84883c4acf00e147f288cee24e91c64c1 WatchSource:0}: Error finding container 2c416e36acbb3c03e43fc53d66c440b84883c4acf00e147f288cee24e91c64c1: Status 404 returned error can't find the container with id 2c416e36acbb3c03e43fc53d66c440b84883c4acf00e147f288cee24e91c64c1 Jul 24 14:19:30.433878 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:30.433840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" event={"ID":"11a139cb-afad-4ec8-b567-541387bbe0c1","Type":"ContainerStarted","Data":"6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7"} Jul 24 14:19:30.434062 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:30.433987 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:30.434062 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:30.434009 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" event={"ID":"11a139cb-afad-4ec8-b567-541387bbe0c1","Type":"ContainerStarted","Data":"2c416e36acbb3c03e43fc53d66c440b84883c4acf00e147f288cee24e91c64c1"} Jul 24 14:19:30.462033 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:30.461971 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" podStartSLOduration=1.461951895 podStartE2EDuration="1.461951895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:19:30.459092319 +0000 UTC m=+2467.069572057" watchObservedRunningTime="2026-07-24 14:19:30.461951895 +0000 UTC m=+2467.072431644" Jul 24 14:19:33.394788 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:33.394712 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" podUID="46afc43a-1618-4d36-bc69-3540964b5783" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.60:9004: connect: connection refused" Jul 24 14:19:41.442213 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:41.442178 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:19:43.393940 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:19:43.393904 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:23:24.179758 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:23:24.179688 2568 scope.go:117] "RemoveContainer" containerID="81f24a3d0c79446a2cf0c19ca3eced60b9c51625ac2cf8a3fdbf91f74c3040f4" Jul 24 14:24:24.200712 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:24:24.200675 2568 scope.go:117] "RemoveContainer" containerID="449f7582bc19235a79e2510798ba5c4f9b0802188cba392573e6f80f1b7f67b7" Jul 24 14:24:24.212117 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:24:24.212091 2568 scope.go:117] "RemoveContainer" containerID="91587129afaea54c5e6fa060553a276eebdde486d04bb55d168f45fb31307492" Jul 24 14:25:16.617938 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.617856 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139958972588464"] Jul 24 14:25:16.621379 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.621359 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:16.624611 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.624584 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fnfwz\"" Jul 24 14:25:16.629621 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.629580 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958972588464"] Jul 24 14:25:16.707712 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.707671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzp9d\" (UniqueName: \"kubernetes.io/projected/27b25287-b7b9-4196-ab53-02314c03b44e-kube-api-access-bzp9d\") pod \"test-curl-7778-139958972588464\" (UID: \"27b25287-b7b9-4196-ab53-02314c03b44e\") " pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:16.808652 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.808613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp9d\" (UniqueName: \"kubernetes.io/projected/27b25287-b7b9-4196-ab53-02314c03b44e-kube-api-access-bzp9d\") pod \"test-curl-7778-139958972588464\" (UID: \"27b25287-b7b9-4196-ab53-02314c03b44e\") " pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:16.818825 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.818780 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp9d\" (UniqueName: \"kubernetes.io/projected/27b25287-b7b9-4196-ab53-02314c03b44e-kube-api-access-bzp9d\") pod \"test-curl-7778-139958972588464\" (UID: \"27b25287-b7b9-4196-ab53-02314c03b44e\") " pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:16.932998 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:16.932890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:17.074868 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:17.074841 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958972588464"] Jul 24 14:25:17.076713 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:17.076687 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27b25287_b7b9_4196_ab53_02314c03b44e.slice/crio-89ec4a483a18b1c7f1491dcafc1386e7c0dea753b6f862dd25a41959c90b18a5 WatchSource:0}: Error finding container 89ec4a483a18b1c7f1491dcafc1386e7c0dea753b6f862dd25a41959c90b18a5: Status 404 returned error can't find the container with id 89ec4a483a18b1c7f1491dcafc1386e7c0dea753b6f862dd25a41959c90b18a5 Jul 24 14:25:17.078087 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:17.078067 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 14:25:17.471608 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:17.471568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958972588464" event={"ID":"27b25287-b7b9-4196-ab53-02314c03b44e","Type":"ContainerStarted","Data":"89ec4a483a18b1c7f1491dcafc1386e7c0dea753b6f862dd25a41959c90b18a5"} Jul 24 14:25:18.479124 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:18.479089 2568 generic.go:356] "Generic (PLEG): container finished" podID="27b25287-b7b9-4196-ab53-02314c03b44e" containerID="704598cc7b9dd1f6b581c5d79c53452119b2730d7863c7d8bb6b860f0c0ab8af" exitCode=0 Jul 24 14:25:18.479555 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:18.479127 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958972588464" event={"ID":"27b25287-b7b9-4196-ab53-02314c03b44e","Type":"ContainerDied","Data":"704598cc7b9dd1f6b581c5d79c53452119b2730d7863c7d8bb6b860f0c0ab8af"} Jul 24 14:25:19.619442 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.619415 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:19.667328 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.667298 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139958972588464_27b25287-b7b9-4196-ab53-02314c03b44e/test-curl-7778-139958972588464/0.log" Jul 24 14:25:19.691817 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.691782 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139958972588464"] Jul 24 14:25:19.699835 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.699805 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139958972588464"] Jul 24 14:25:19.738520 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.738438 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzp9d\" (UniqueName: \"kubernetes.io/projected/27b25287-b7b9-4196-ab53-02314c03b44e-kube-api-access-bzp9d\") pod \"27b25287-b7b9-4196-ab53-02314c03b44e\" (UID: \"27b25287-b7b9-4196-ab53-02314c03b44e\") " Jul 24 14:25:19.740658 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.740629 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b25287-b7b9-4196-ab53-02314c03b44e-kube-api-access-bzp9d" (OuterVolumeSpecName: "kube-api-access-bzp9d") pod "27b25287-b7b9-4196-ab53-02314c03b44e" (UID: "27b25287-b7b9-4196-ab53-02314c03b44e"). InnerVolumeSpecName "kube-api-access-bzp9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:25:19.826099 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.826049 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139958972590128"] Jul 24 14:25:19.826492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.826476 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b25287-b7b9-4196-ab53-02314c03b44e" containerName="test-curl-7778-139958972588464" Jul 24 14:25:19.826538 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.826494 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b25287-b7b9-4196-ab53-02314c03b44e" containerName="test-curl-7778-139958972588464" Jul 24 14:25:19.826571 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.826560 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b25287-b7b9-4196-ab53-02314c03b44e" containerName="test-curl-7778-139958972588464" Jul 24 14:25:19.833865 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.831971 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:19.839300 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.839270 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzp9d\" (UniqueName: \"kubernetes.io/projected/27b25287-b7b9-4196-ab53-02314c03b44e-kube-api-access-bzp9d\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:25:19.840444 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.840417 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958972590128"] Jul 24 14:25:19.860831 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.860799 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b25287-b7b9-4196-ab53-02314c03b44e" path="/var/lib/kubelet/pods/27b25287-b7b9-4196-ab53-02314c03b44e/volumes" Jul 24 14:25:19.940584 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:19.940540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhng\" (UniqueName: \"kubernetes.io/projected/e5c599f4-09cb-4fbb-af13-9598a3e6d999-kube-api-access-plhng\") pod \"test-curl-7778-139958972590128\" (UID: \"e5c599f4-09cb-4fbb-af13-9598a3e6d999\") " pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:20.041588 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.041480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhng\" (UniqueName: \"kubernetes.io/projected/e5c599f4-09cb-4fbb-af13-9598a3e6d999-kube-api-access-plhng\") pod \"test-curl-7778-139958972590128\" (UID: \"e5c599f4-09cb-4fbb-af13-9598a3e6d999\") " pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:20.051589 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.051559 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhng\" (UniqueName: \"kubernetes.io/projected/e5c599f4-09cb-4fbb-af13-9598a3e6d999-kube-api-access-plhng\") pod \"test-curl-7778-139958972590128\" (UID: \"e5c599f4-09cb-4fbb-af13-9598a3e6d999\") " pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:20.145218 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.145183 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:20.288787 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.288702 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958972590128"] Jul 24 14:25:20.291562 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:20.291528 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c599f4_09cb_4fbb_af13_9598a3e6d999.slice/crio-78dce8fc9fdabea0d722e4db60926cf28db53c78c7c48cbb300dc07c59ec9950 WatchSource:0}: Error finding container 78dce8fc9fdabea0d722e4db60926cf28db53c78c7c48cbb300dc07c59ec9950: Status 404 returned error can't find the container with id 78dce8fc9fdabea0d722e4db60926cf28db53c78c7c48cbb300dc07c59ec9950 Jul 24 14:25:20.491474 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.491435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958972590128" event={"ID":"e5c599f4-09cb-4fbb-af13-9598a3e6d999","Type":"ContainerStarted","Data":"78dce8fc9fdabea0d722e4db60926cf28db53c78c7c48cbb300dc07c59ec9950"} Jul 24 14:25:20.492776 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.492751 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972588464" Jul 24 14:25:20.492908 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:20.492791 2568 scope.go:117] "RemoveContainer" containerID="704598cc7b9dd1f6b581c5d79c53452119b2730d7863c7d8bb6b860f0c0ab8af" Jul 24 14:25:21.499970 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:21.499937 2568 generic.go:356] "Generic (PLEG): container finished" podID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" containerID="24cc5ddf47e0179d971650e9d1c094e26c9b8a428692338817e5cba274b34fe8" exitCode=0 Jul 24 14:25:21.500389 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:21.500025 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958972590128" event={"ID":"e5c599f4-09cb-4fbb-af13-9598a3e6d999","Type":"ContainerDied","Data":"24cc5ddf47e0179d971650e9d1c094e26c9b8a428692338817e5cba274b34fe8"} Jul 24 14:25:22.649408 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.649381 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:22.674717 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.674688 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139958972590128_e5c599f4-09cb-4fbb-af13-9598a3e6d999/test-curl-7778-139958972590128/0.log" Jul 24 14:25:22.704193 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.704152 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139958972590128"] Jul 24 14:25:22.708903 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.708875 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139958972590128"] Jul 24 14:25:22.770062 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.769961 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plhng\" (UniqueName: \"kubernetes.io/projected/e5c599f4-09cb-4fbb-af13-9598a3e6d999-kube-api-access-plhng\") pod \"e5c599f4-09cb-4fbb-af13-9598a3e6d999\" (UID: \"e5c599f4-09cb-4fbb-af13-9598a3e6d999\") " Jul 24 14:25:22.772126 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.772101 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c599f4-09cb-4fbb-af13-9598a3e6d999-kube-api-access-plhng" (OuterVolumeSpecName: "kube-api-access-plhng") pod "e5c599f4-09cb-4fbb-af13-9598a3e6d999" (UID: "e5c599f4-09cb-4fbb-af13-9598a3e6d999"). InnerVolumeSpecName "kube-api-access-plhng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:25:22.870888 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.870853 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plhng\" (UniqueName: \"kubernetes.io/projected/e5c599f4-09cb-4fbb-af13-9598a3e6d999-kube-api-access-plhng\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:25:22.883687 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.883654 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139958971850592"] Jul 24 14:25:22.884090 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.884074 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" containerName="test-curl-7778-139958972590128" Jul 24 14:25:22.884141 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.884093 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" containerName="test-curl-7778-139958972590128" Jul 24 14:25:22.884177 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.884166 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" containerName="test-curl-7778-139958972590128" Jul 24 14:25:22.887197 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.887179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:22.896022 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.895994 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958971850592"] Jul 24 14:25:22.972108 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:22.972067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ssr\" (UniqueName: \"kubernetes.io/projected/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1-kube-api-access-c7ssr\") pod \"test-curl-7778-139958971850592\" (UID: \"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1\") " pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:23.073701 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.073604 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ssr\" (UniqueName: \"kubernetes.io/projected/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1-kube-api-access-c7ssr\") pod \"test-curl-7778-139958971850592\" (UID: \"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1\") " pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:23.083529 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.083502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ssr\" (UniqueName: \"kubernetes.io/projected/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1-kube-api-access-c7ssr\") pod \"test-curl-7778-139958971850592\" (UID: \"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1\") " pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:23.198614 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.198573 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:23.344437 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.344412 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958971850592"] Jul 24 14:25:23.346327 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:23.346297 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c755db2_3fdb_4fb4_9d74_43d7589e0bf1.slice/crio-d3457b25933b3b299a1d6d18b017778447cc13729aa32434bbf65d9aac6999e2 WatchSource:0}: Error finding container d3457b25933b3b299a1d6d18b017778447cc13729aa32434bbf65d9aac6999e2: Status 404 returned error can't find the container with id d3457b25933b3b299a1d6d18b017778447cc13729aa32434bbf65d9aac6999e2 Jul 24 14:25:23.514055 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.514026 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:23.514239 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.514075 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78dce8fc9fdabea0d722e4db60926cf28db53c78c7c48cbb300dc07c59ec9950" Jul 24 14:25:23.515347 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.515312 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958971850592" event={"ID":"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1","Type":"ContainerStarted","Data":"d3457b25933b3b299a1d6d18b017778447cc13729aa32434bbf65d9aac6999e2"} Jul 24 14:25:23.525367 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:25:23.525336 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139958972590128\" is forbidden: User \"system:node:ip-10-0-138-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-248.ec2.internal' and this object" podUID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:23.859123 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:25:23.859077 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139958972590128\" is forbidden: User \"system:node:ip-10-0-138-248.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-248.ec2.internal' and this object" podUID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" pod="opendatahub/test-curl-7778-139958972590128" Jul 24 14:25:23.860708 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:23.860677 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c599f4-09cb-4fbb-af13-9598a3e6d999" path="/var/lib/kubelet/pods/e5c599f4-09cb-4fbb-af13-9598a3e6d999/volumes" Jul 24 14:25:24.237624 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:24.237585 2568 scope.go:117] "RemoveContainer" containerID="61305ee14ff227e27c26041401bf3b1e945c5d40444547870be22a12c0fd25c6" Jul 24 14:25:24.529097 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:24.528998 2568 generic.go:356] "Generic (PLEG): container finished" podID="8c755db2-3fdb-4fb4-9d74-43d7589e0bf1" containerID="e56a220aa040ef3a3fbde27b54d7ea3c1618e0cdf57dcbf9c97b1aca92bd0b35" exitCode=0 Jul 24 14:25:24.529253 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:24.529093 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958971850592" event={"ID":"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1","Type":"ContainerDied","Data":"e56a220aa040ef3a3fbde27b54d7ea3c1618e0cdf57dcbf9c97b1aca92bd0b35"} Jul 24 14:25:25.672642 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.672613 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:25.695366 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.695333 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139958971850592_8c755db2-3fdb-4fb4-9d74-43d7589e0bf1/test-curl-7778-139958971850592/0.log" Jul 24 14:25:25.722416 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.722376 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139958971850592"] Jul 24 14:25:25.727492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.727458 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139958971850592"] Jul 24 14:25:25.801707 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.801606 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7ssr\" (UniqueName: \"kubernetes.io/projected/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1-kube-api-access-c7ssr\") pod \"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1\" (UID: \"8c755db2-3fdb-4fb4-9d74-43d7589e0bf1\") " Jul 24 14:25:25.803813 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.803782 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1-kube-api-access-c7ssr" (OuterVolumeSpecName: "kube-api-access-c7ssr") pod "8c755db2-3fdb-4fb4-9d74-43d7589e0bf1" (UID: "8c755db2-3fdb-4fb4-9d74-43d7589e0bf1"). InnerVolumeSpecName "kube-api-access-c7ssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:25:25.860693 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.860658 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c755db2-3fdb-4fb4-9d74-43d7589e0bf1" path="/var/lib/kubelet/pods/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1/volumes" Jul 24 14:25:25.903541 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:25.903499 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ssr\" (UniqueName: \"kubernetes.io/projected/8c755db2-3fdb-4fb4-9d74-43d7589e0bf1-kube-api-access-c7ssr\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:25:26.541824 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:26.541790 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971850592" Jul 24 14:25:26.542017 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:26.541836 2568 scope.go:117] "RemoveContainer" containerID="e56a220aa040ef3a3fbde27b54d7ea3c1618e0cdf57dcbf9c97b1aca92bd0b35" Jul 24 14:25:29.346027 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.345990 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139958971889104"] Jul 24 14:25:29.346426 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.346407 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c755db2-3fdb-4fb4-9d74-43d7589e0bf1" containerName="test-curl-7778-139958971850592" Jul 24 14:25:29.346426 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.346420 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c755db2-3fdb-4fb4-9d74-43d7589e0bf1" containerName="test-curl-7778-139958971850592" Jul 24 14:25:29.346516 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.346504 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c755db2-3fdb-4fb4-9d74-43d7589e0bf1" containerName="test-curl-7778-139958971850592" Jul 24 14:25:29.351164 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.351144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:29.354074 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.354048 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fnfwz\"" Jul 24 14:25:29.355140 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.355109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958971889104"] Jul 24 14:25:29.435357 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.435325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfls\" (UniqueName: \"kubernetes.io/projected/040a2600-05cf-489f-8f82-b262f02c3150-kube-api-access-dcfls\") pod \"test-curl-7778-139958971889104\" (UID: \"040a2600-05cf-489f-8f82-b262f02c3150\") " pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:29.536139 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.536096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfls\" (UniqueName: \"kubernetes.io/projected/040a2600-05cf-489f-8f82-b262f02c3150-kube-api-access-dcfls\") pod \"test-curl-7778-139958971889104\" (UID: \"040a2600-05cf-489f-8f82-b262f02c3150\") " pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:29.545405 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.545376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfls\" (UniqueName: \"kubernetes.io/projected/040a2600-05cf-489f-8f82-b262f02c3150-kube-api-access-dcfls\") pod \"test-curl-7778-139958971889104\" (UID: \"040a2600-05cf-489f-8f82-b262f02c3150\") " pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:29.661398 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.661353 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:29.814251 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:29.814221 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139958971889104"] Jul 24 14:25:29.816397 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:29.816368 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod040a2600_05cf_489f_8f82_b262f02c3150.slice/crio-282ea83a317abe3b669b5ab32c0cca6ffd8ef735deeaa24a33b615c8ee4c8a6c WatchSource:0}: Error finding container 282ea83a317abe3b669b5ab32c0cca6ffd8ef735deeaa24a33b615c8ee4c8a6c: Status 404 returned error can't find the container with id 282ea83a317abe3b669b5ab32c0cca6ffd8ef735deeaa24a33b615c8ee4c8a6c Jul 24 14:25:30.566797 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:30.566754 2568 generic.go:356] "Generic (PLEG): container finished" podID="040a2600-05cf-489f-8f82-b262f02c3150" containerID="869503ee456baabf6ae1b73b5b034c73a337aeb72e3003bfeee7932f17e44c7e" exitCode=0 Jul 24 14:25:30.567220 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:30.566836 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958971889104" event={"ID":"040a2600-05cf-489f-8f82-b262f02c3150","Type":"ContainerDied","Data":"869503ee456baabf6ae1b73b5b034c73a337aeb72e3003bfeee7932f17e44c7e"} Jul 24 14:25:30.567220 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:30.566879 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139958971889104" event={"ID":"040a2600-05cf-489f-8f82-b262f02c3150","Type":"ContainerStarted","Data":"282ea83a317abe3b669b5ab32c0cca6ffd8ef735deeaa24a33b615c8ee4c8a6c"} Jul 24 14:25:31.715342 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.715316 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:31.737349 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.737317 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139958971889104_040a2600-05cf-489f-8f82-b262f02c3150/test-curl-7778-139958971889104/0.log" Jul 24 14:25:31.755023 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.754984 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcfls\" (UniqueName: \"kubernetes.io/projected/040a2600-05cf-489f-8f82-b262f02c3150-kube-api-access-dcfls\") pod \"040a2600-05cf-489f-8f82-b262f02c3150\" (UID: \"040a2600-05cf-489f-8f82-b262f02c3150\") " Jul 24 14:25:31.757120 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.757079 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040a2600-05cf-489f-8f82-b262f02c3150-kube-api-access-dcfls" (OuterVolumeSpecName: "kube-api-access-dcfls") pod "040a2600-05cf-489f-8f82-b262f02c3150" (UID: "040a2600-05cf-489f-8f82-b262f02c3150"). InnerVolumeSpecName "kube-api-access-dcfls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:25:31.768703 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.768672 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139958971889104"] Jul 24 14:25:31.772850 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.772824 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139958971889104"] Jul 24 14:25:31.856624 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.856535 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfls\" (UniqueName: \"kubernetes.io/projected/040a2600-05cf-489f-8f82-b262f02c3150-kube-api-access-dcfls\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:25:31.860334 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:31.860309 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="040a2600-05cf-489f-8f82-b262f02c3150" path="/var/lib/kubelet/pods/040a2600-05cf-489f-8f82-b262f02c3150/volumes" Jul 24 14:25:32.580005 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:32.579975 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139958971889104" Jul 24 14:25:32.580207 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:32.580026 2568 scope.go:117] "RemoveContainer" containerID="869503ee456baabf6ae1b73b5b034c73a337aeb72e3003bfeee7932f17e44c7e" Jul 24 14:25:39.002338 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.002298 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq"] Jul 24 14:25:39.002792 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.002759 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="040a2600-05cf-489f-8f82-b262f02c3150" containerName="test-curl-7778-139958971889104" Jul 24 14:25:39.002792 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.002774 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="040a2600-05cf-489f-8f82-b262f02c3150" containerName="test-curl-7778-139958971889104" Jul 24 14:25:39.002874 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.002864 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="040a2600-05cf-489f-8f82-b262f02c3150" containerName="test-curl-7778-139958971889104" Jul 24 14:25:39.007890 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.007864 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.010545 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.010523 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-15e86b\"" Jul 24 14:25:39.015970 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.015940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq"] Jul 24 14:25:39.043634 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.043587 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h"] Jul 24 14:25:39.048378 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.048352 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.051260 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.051236 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-15e86b-dockercfg-dg585\"" Jul 24 14:25:39.056116 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.056085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h"] Jul 24 14:25:39.129065 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.129021 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrdbw\" (UniqueName: \"kubernetes.io/projected/0d3993a2-1937-4dc1-a193-150b2acf3976-kube-api-access-wrdbw\") pod \"payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.129236 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.129075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0757972a-a9db-418e-9b27-756823cbf84d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.129236 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.129206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3993a2-1937-4dc1-a193-150b2acf3976-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.129352 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.129276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4lc\" (UniqueName: \"kubernetes.io/projected/0757972a-a9db-418e-9b27-756823cbf84d-kube-api-access-xs4lc\") pod \"payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.230869 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.230828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4lc\" (UniqueName: \"kubernetes.io/projected/0757972a-a9db-418e-9b27-756823cbf84d-kube-api-access-xs4lc\") pod \"payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.231050 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.230940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrdbw\" (UniqueName: \"kubernetes.io/projected/0d3993a2-1937-4dc1-a193-150b2acf3976-kube-api-access-wrdbw\") pod \"payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.231050 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.230975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0757972a-a9db-418e-9b27-756823cbf84d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.231050 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.231039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3993a2-1937-4dc1-a193-150b2acf3976-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.231794 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.231776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0757972a-a9db-418e-9b27-756823cbf84d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.231838 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.231817 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3993a2-1937-4dc1-a193-150b2acf3976-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.239834 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.239804 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4lc\" (UniqueName: \"kubernetes.io/projected/0757972a-a9db-418e-9b27-756823cbf84d-kube-api-access-xs4lc\") pod \"payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.239976 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.239938 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrdbw\" (UniqueName: \"kubernetes.io/projected/0d3993a2-1937-4dc1-a193-150b2acf3976-kube-api-access-wrdbw\") pod \"payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.320717 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.320622 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.361718 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.361674 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:39.492503 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.492410 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq"] Jul 24 14:25:39.502948 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:39.502915 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3993a2_1937_4dc1_a193_150b2acf3976.slice/crio-55866040ef58b03c8983011090d44b3304eac1ef963cd34229f6f85462bc38ae WatchSource:0}: Error finding container 55866040ef58b03c8983011090d44b3304eac1ef963cd34229f6f85462bc38ae: Status 404 returned error can't find the container with id 55866040ef58b03c8983011090d44b3304eac1ef963cd34229f6f85462bc38ae Jul 24 14:25:39.554041 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.554017 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h"] Jul 24 14:25:39.556001 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:39.555977 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0757972a_a9db_418e_9b27_756823cbf84d.slice/crio-a023029a886b9ed06015cbf02cb838f638584c00bfeb2d6c7c56e974fd7418bc WatchSource:0}: Error finding container a023029a886b9ed06015cbf02cb838f638584c00bfeb2d6c7c56e974fd7418bc: Status 404 returned error can't find the container with id a023029a886b9ed06015cbf02cb838f638584c00bfeb2d6c7c56e974fd7418bc Jul 24 14:25:39.622626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.622593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" event={"ID":"0d3993a2-1937-4dc1-a193-150b2acf3976","Type":"ContainerStarted","Data":"bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74"} Jul 24 14:25:39.622781 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.622654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" event={"ID":"0d3993a2-1937-4dc1-a193-150b2acf3976","Type":"ContainerStarted","Data":"55866040ef58b03c8983011090d44b3304eac1ef963cd34229f6f85462bc38ae"} Jul 24 14:25:39.622781 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.622685 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:39.624607 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.624581 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" event={"ID":"0757972a-a9db-418e-9b27-756823cbf84d","Type":"ContainerStarted","Data":"a023029a886b9ed06015cbf02cb838f638584c00bfeb2d6c7c56e974fd7418bc"} Jul 24 14:25:39.641667 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:39.641602 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" podStartSLOduration=1.641583957 podStartE2EDuration="1.641583957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:39.638857961 +0000 UTC m=+2836.249337711" watchObservedRunningTime="2026-07-24 14:25:39.641583957 +0000 UTC m=+2836.252063703" Jul 24 14:25:40.633116 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:40.633076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" event={"ID":"0757972a-a9db-418e-9b27-756823cbf84d","Type":"ContainerStarted","Data":"a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94"} Jul 24 14:25:40.633604 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:40.633168 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:40.657011 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:40.656958 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" podStartSLOduration=1.656938293 podStartE2EDuration="1.656938293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:40.654683514 +0000 UTC m=+2837.265163262" watchObservedRunningTime="2026-07-24 14:25:40.656938293 +0000 UTC m=+2837.267418038" Jul 24 14:25:50.635506 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:50.635471 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:25:51.642175 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:51.642144 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:25:57.324154 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.324119 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc"] Jul 24 14:25:57.328338 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.328310 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.331470 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.331428 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-03c962-serving-cert\"" Jul 24 14:25:57.341650 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.341618 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc"] Jul 24 14:25:57.416299 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.416256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shwf\" (UniqueName: \"kubernetes.io/projected/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-kube-api-access-2shwf\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.416481 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.416387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.416481 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.416469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.517606 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.517569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.517825 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.517625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shwf\" (UniqueName: \"kubernetes.io/projected/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-kube-api-access-2shwf\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.517825 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.517688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.517905 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:25:57.517837 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-serving-cert: secret "maas-api-e2e-ipp-b-03c962-serving-cert" not found Jul 24 14:25:57.517940 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:25:57.517905 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls podName:2a814ae4-15a6-4cab-9b29-fedc0bb58bb4 nodeName:}" failed. No retries permitted until 2026-07-24 14:25:58.017887571 +0000 UTC m=+2854.628367304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls") pod "maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" (UID: "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4") : secret "maas-api-e2e-ipp-b-03c962-serving-cert" not found Jul 24 14:25:57.518330 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.518308 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:57.526702 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:57.526671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shwf\" (UniqueName: \"kubernetes.io/projected/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-kube-api-access-2shwf\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:58.023026 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:58.022990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:58.025627 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:58.025601 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:58.241404 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:58.241363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:58.392754 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:58.392688 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc"] Jul 24 14:25:58.395048 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:25:58.395021 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a814ae4_15a6_4cab_9b29_fedc0bb58bb4.slice/crio-8c7f0cad2ae2393f0413d5050f50286cb389ef0dc778fe522baaed38b22b77d8 WatchSource:0}: Error finding container 8c7f0cad2ae2393f0413d5050f50286cb389ef0dc778fe522baaed38b22b77d8: Status 404 returned error can't find the container with id 8c7f0cad2ae2393f0413d5050f50286cb389ef0dc778fe522baaed38b22b77d8 Jul 24 14:25:58.736542 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:58.736505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" event={"ID":"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4","Type":"ContainerStarted","Data":"8c7f0cad2ae2393f0413d5050f50286cb389ef0dc778fe522baaed38b22b77d8"} Jul 24 14:25:59.744310 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:59.744272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" event={"ID":"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4","Type":"ContainerStarted","Data":"915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27"} Jul 24 14:25:59.744715 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:59.744388 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:25:59.765133 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:25:59.765081 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" podStartSLOduration=2.3532090070000002 podStartE2EDuration="2.765065045s" totalImagesPullingTime="411.856038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:25:59.763787501 +0000 UTC m=+2856.374267245" watchObservedRunningTime="2026-07-24 14:25:59.765065045 +0000 UTC m=+2856.375544791" Jul 24 14:26:05.755798 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:26:05.755763 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:26:56.041266 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:26:56.041238 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h_0757972a-a9db-418e-9b27-756823cbf84d/payload-processing/0.log" Jul 24 14:27:00.307606 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.307565 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg"] Jul 24 14:27:00.311459 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.311439 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.315232 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.315205 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d95e17-serving-cert\"" Jul 24 14:27:00.332352 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.332319 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg"] Jul 24 14:27:00.429923 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.429883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e529802e-2728-40fa-839f-85e8a113ba55-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.430104 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.430033 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgsrb\" (UniqueName: \"kubernetes.io/projected/e529802e-2728-40fa-839f-85e8a113ba55-kube-api-access-zgsrb\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.430104 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.430082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e529802e-2728-40fa-839f-85e8a113ba55-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.531141 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.531105 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e529802e-2728-40fa-839f-85e8a113ba55-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.531365 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.531231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgsrb\" (UniqueName: \"kubernetes.io/projected/e529802e-2728-40fa-839f-85e8a113ba55-kube-api-access-zgsrb\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.531365 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.531266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e529802e-2728-40fa-839f-85e8a113ba55-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.531949 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.531924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e529802e-2728-40fa-839f-85e8a113ba55-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.533620 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.533595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e529802e-2728-40fa-839f-85e8a113ba55-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.545734 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.545696 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgsrb\" (UniqueName: \"kubernetes.io/projected/e529802e-2728-40fa-839f-85e8a113ba55-kube-api-access-zgsrb\") pod \"maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.621701 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.621668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:00.769154 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:00.769125 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg"] Jul 24 14:27:00.771206 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:27:00.771170 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode529802e_2728_40fa_839f_85e8a113ba55.slice/crio-b2046a6fe763b48c7ce01719570dc160cf630285a377d76dbfb51c777b6df32b WatchSource:0}: Error finding container b2046a6fe763b48c7ce01719570dc160cf630285a377d76dbfb51c777b6df32b: Status 404 returned error can't find the container with id b2046a6fe763b48c7ce01719570dc160cf630285a377d76dbfb51c777b6df32b Jul 24 14:27:01.099165 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.099076 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" event={"ID":"e529802e-2728-40fa-839f-85e8a113ba55","Type":"ContainerStarted","Data":"b2046a6fe763b48c7ce01719570dc160cf630285a377d76dbfb51c777b6df32b"} Jul 24 14:27:01.127934 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.127898 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p"] Jul 24 14:27:01.132687 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.132667 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.135742 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.135710 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d95e17\"" Jul 24 14:27:01.135895 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.135748 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d95e17-dockercfg-28vdd\"" Jul 24 14:27:01.142367 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.142339 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p"] Jul 24 14:27:01.238594 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.238557 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8lz8\" (UniqueName: \"kubernetes.io/projected/53f9af35-e075-4687-b43f-e511a8a4d774-kube-api-access-r8lz8\") pod \"payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.238824 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.238797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f9af35-e075-4687-b43f-e511a8a4d774-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.339507 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.339474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8lz8\" (UniqueName: \"kubernetes.io/projected/53f9af35-e075-4687-b43f-e511a8a4d774-kube-api-access-r8lz8\") pod \"payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.339953 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.339579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f9af35-e075-4687-b43f-e511a8a4d774-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.340289 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.340268 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f9af35-e075-4687-b43f-e511a8a4d774-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.348913 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.348882 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8lz8\" (UniqueName: \"kubernetes.io/projected/53f9af35-e075-4687-b43f-e511a8a4d774-kube-api-access-r8lz8\") pod \"payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.445410 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.445370 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:01.592845 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:01.592814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p"] Jul 24 14:27:01.594563 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:27:01.594534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f9af35_e075_4687_b43f_e511a8a4d774.slice/crio-c7b0a1372ef83f5c1199a26c95b0168d650f98504d394ea215cd348730771532 WatchSource:0}: Error finding container c7b0a1372ef83f5c1199a26c95b0168d650f98504d394ea215cd348730771532: Status 404 returned error can't find the container with id c7b0a1372ef83f5c1199a26c95b0168d650f98504d394ea215cd348730771532 Jul 24 14:27:02.114420 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:02.114379 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" event={"ID":"53f9af35-e075-4687-b43f-e511a8a4d774","Type":"ContainerStarted","Data":"5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33"} Jul 24 14:27:02.114633 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:02.114615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:02.114713 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:02.114652 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" event={"ID":"53f9af35-e075-4687-b43f-e511a8a4d774","Type":"ContainerStarted","Data":"c7b0a1372ef83f5c1199a26c95b0168d650f98504d394ea215cd348730771532"} Jul 24 14:27:02.116030 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:02.116004 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" event={"ID":"e529802e-2728-40fa-839f-85e8a113ba55","Type":"ContainerStarted","Data":"ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0"} Jul 24 14:27:02.116162 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:02.116141 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:02.136626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:02.136565 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" podStartSLOduration=1.1365487619999999 podStartE2EDuration="1.136548762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:27:02.13382677 +0000 UTC m=+2918.744306519" watchObservedRunningTime="2026-07-24 14:27:02.136548762 +0000 UTC m=+2918.747028537" Jul 24 14:27:08.127606 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:08.127575 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:08.147165 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:08.147099 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" podStartSLOduration=7.634165961 podStartE2EDuration="8.147081545s" totalImagesPullingTime="512.915584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:27:02.154256811 +0000 UTC m=+2918.764736553" watchObservedRunningTime="2026-07-24 14:27:08.147081545 +0000 UTC m=+2924.757561290" Jul 24 14:27:13.124352 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:13.124318 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:15.767421 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.767377 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt"] Jul 24 14:27:15.771170 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.771153 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:15.777069 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.777035 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:27:15.783475 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.783446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt"] Jul 24 14:27:15.887332 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.887283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd07165-8665-4500-9e44-cf56a7cbad74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt\" (UID: \"2fd07165-8665-4500-9e44-cf56a7cbad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:15.989166 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.989120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd07165-8665-4500-9e44-cf56a7cbad74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt\" (UID: \"2fd07165-8665-4500-9e44-cf56a7cbad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:15.989886 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:15.989866 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd07165-8665-4500-9e44-cf56a7cbad74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt\" (UID: \"2fd07165-8665-4500-9e44-cf56a7cbad74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:16.082857 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:16.082764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:16.227458 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:16.227423 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt"] Jul 24 14:27:16.228587 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:27:16.228563 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd07165_8665_4500_9e44_cf56a7cbad74.slice/crio-cb77040c5c92b6d3c85256a789713b3a37d64800109ad77bddb027acd6bdbc18 WatchSource:0}: Error finding container cb77040c5c92b6d3c85256a789713b3a37d64800109ad77bddb027acd6bdbc18: Status 404 returned error can't find the container with id cb77040c5c92b6d3c85256a789713b3a37d64800109ad77bddb027acd6bdbc18 Jul 24 14:27:17.208437 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:17.208399 2568 generic.go:356] "Generic (PLEG): container finished" podID="2fd07165-8665-4500-9e44-cf56a7cbad74" containerID="810db5bb29b09a62db8ad81583d41fcc62fa779fd6cd1e00df725376eeef7e8c" exitCode=0 Jul 24 14:27:17.208852 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:17.208490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" event={"ID":"2fd07165-8665-4500-9e44-cf56a7cbad74","Type":"ContainerDied","Data":"810db5bb29b09a62db8ad81583d41fcc62fa779fd6cd1e00df725376eeef7e8c"} Jul 24 14:27:17.208852 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:17.208545 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" event={"ID":"2fd07165-8665-4500-9e44-cf56a7cbad74","Type":"ContainerStarted","Data":"cb77040c5c92b6d3c85256a789713b3a37d64800109ad77bddb027acd6bdbc18"} Jul 24 14:27:18.357911 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:18.357881 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:18.511434 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:18.511315 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd07165-8665-4500-9e44-cf56a7cbad74-maas-api-service-ca\") pod \"2fd07165-8665-4500-9e44-cf56a7cbad74\" (UID: \"2fd07165-8665-4500-9e44-cf56a7cbad74\") " Jul 24 14:27:18.511689 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:18.511664 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd07165-8665-4500-9e44-cf56a7cbad74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fd07165-8665-4500-9e44-cf56a7cbad74" (UID: "2fd07165-8665-4500-9e44-cf56a7cbad74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:18.612283 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:18.612229 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fd07165-8665-4500-9e44-cf56a7cbad74-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:19.228000 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:19.227969 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" Jul 24 14:27:19.228000 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:19.227981 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt" event={"ID":"2fd07165-8665-4500-9e44-cf56a7cbad74","Type":"ContainerDied","Data":"cb77040c5c92b6d3c85256a789713b3a37d64800109ad77bddb027acd6bdbc18"} Jul 24 14:27:19.228217 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:19.228053 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb77040c5c92b6d3c85256a789713b3a37d64800109ad77bddb027acd6bdbc18" Jul 24 14:27:26.309034 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.308997 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg"] Jul 24 14:27:26.309492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.309263 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" podUID="e529802e-2728-40fa-839f-85e8a113ba55" containerName="maas-api" containerID="cri-o://ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0" gracePeriod=30 Jul 24 14:27:26.375561 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.375521 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p"] Jul 24 14:27:26.375807 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.375756 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" podUID="53f9af35-e075-4687-b43f-e511a8a4d774" containerName="payload-processing" containerID="cri-o://5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33" gracePeriod=30 Jul 24 14:27:26.676216 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.676181 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:26.683900 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.683876 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:26.794659 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.794620 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f9af35-e075-4687-b43f-e511a8a4d774-plugins-config-volume\") pod \"53f9af35-e075-4687-b43f-e511a8a4d774\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " Jul 24 14:27:26.794843 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.794673 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e529802e-2728-40fa-839f-85e8a113ba55-maas-api-tls\") pod \"e529802e-2728-40fa-839f-85e8a113ba55\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " Jul 24 14:27:26.794843 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.794787 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e529802e-2728-40fa-839f-85e8a113ba55-openshift-service-ca\") pod \"e529802e-2728-40fa-839f-85e8a113ba55\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " Jul 24 14:27:26.794843 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.794817 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgsrb\" (UniqueName: \"kubernetes.io/projected/e529802e-2728-40fa-839f-85e8a113ba55-kube-api-access-zgsrb\") pod \"e529802e-2728-40fa-839f-85e8a113ba55\" (UID: \"e529802e-2728-40fa-839f-85e8a113ba55\") " Jul 24 14:27:26.794843 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.794841 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8lz8\" (UniqueName: \"kubernetes.io/projected/53f9af35-e075-4687-b43f-e511a8a4d774-kube-api-access-r8lz8\") pod \"53f9af35-e075-4687-b43f-e511a8a4d774\" (UID: \"53f9af35-e075-4687-b43f-e511a8a4d774\") " Jul 24 14:27:26.795118 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.795090 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f9af35-e075-4687-b43f-e511a8a4d774-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "53f9af35-e075-4687-b43f-e511a8a4d774" (UID: "53f9af35-e075-4687-b43f-e511a8a4d774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:26.795187 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.795145 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e529802e-2728-40fa-839f-85e8a113ba55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e529802e-2728-40fa-839f-85e8a113ba55" (UID: "e529802e-2728-40fa-839f-85e8a113ba55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:26.795251 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.795237 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53f9af35-e075-4687-b43f-e511a8a4d774-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:26.795307 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.795252 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e529802e-2728-40fa-839f-85e8a113ba55-openshift-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:26.796926 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.796897 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f9af35-e075-4687-b43f-e511a8a4d774-kube-api-access-r8lz8" (OuterVolumeSpecName: "kube-api-access-r8lz8") pod "53f9af35-e075-4687-b43f-e511a8a4d774" (UID: "53f9af35-e075-4687-b43f-e511a8a4d774"). InnerVolumeSpecName "kube-api-access-r8lz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:26.797033 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.796982 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e529802e-2728-40fa-839f-85e8a113ba55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e529802e-2728-40fa-839f-85e8a113ba55" (UID: "e529802e-2728-40fa-839f-85e8a113ba55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:27:26.797033 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.797025 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e529802e-2728-40fa-839f-85e8a113ba55-kube-api-access-zgsrb" (OuterVolumeSpecName: "kube-api-access-zgsrb") pod "e529802e-2728-40fa-839f-85e8a113ba55" (UID: "e529802e-2728-40fa-839f-85e8a113ba55"). InnerVolumeSpecName "kube-api-access-zgsrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:26.896560 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.896525 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e529802e-2728-40fa-839f-85e8a113ba55-maas-api-tls\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:26.896560 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.896559 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgsrb\" (UniqueName: \"kubernetes.io/projected/e529802e-2728-40fa-839f-85e8a113ba55-kube-api-access-zgsrb\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:26.896811 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:26.896570 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8lz8\" (UniqueName: \"kubernetes.io/projected/53f9af35-e075-4687-b43f-e511a8a4d774-kube-api-access-r8lz8\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:27.279314 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.279216 2568 generic.go:356] "Generic (PLEG): container finished" podID="e529802e-2728-40fa-839f-85e8a113ba55" containerID="ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0" exitCode=0 Jul 24 14:27:27.279314 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.279265 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" Jul 24 14:27:27.279666 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.279305 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" event={"ID":"e529802e-2728-40fa-839f-85e8a113ba55","Type":"ContainerDied","Data":"ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0"} Jul 24 14:27:27.279666 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.279372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg" event={"ID":"e529802e-2728-40fa-839f-85e8a113ba55","Type":"ContainerDied","Data":"b2046a6fe763b48c7ce01719570dc160cf630285a377d76dbfb51c777b6df32b"} Jul 24 14:27:27.279666 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.279401 2568 scope.go:117] "RemoveContainer" containerID="ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0" Jul 24 14:27:27.280770 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.280745 2568 generic.go:356] "Generic (PLEG): container finished" podID="53f9af35-e075-4687-b43f-e511a8a4d774" containerID="5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33" exitCode=0 Jul 24 14:27:27.280943 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.280780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" event={"ID":"53f9af35-e075-4687-b43f-e511a8a4d774","Type":"ContainerDied","Data":"5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33"} Jul 24 14:27:27.280943 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.280794 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" Jul 24 14:27:27.280943 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.280833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p" event={"ID":"53f9af35-e075-4687-b43f-e511a8a4d774","Type":"ContainerDied","Data":"c7b0a1372ef83f5c1199a26c95b0168d650f98504d394ea215cd348730771532"} Jul 24 14:27:27.289555 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.289515 2568 scope.go:117] "RemoveContainer" containerID="ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0" Jul 24 14:27:27.289882 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:27:27.289860 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0\": container with ID starting with ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0 not found: ID does not exist" containerID="ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0" Jul 24 14:27:27.289986 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.289891 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0"} err="failed to get container status \"ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0\": rpc error: code = NotFound desc = could not find container \"ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0\": container with ID starting with ccc716df4075d28e05ded7e664d07c7d39af076baf023ccea65cec9a6a22a2a0 not found: ID does not exist" Jul 24 14:27:27.289986 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.289909 2568 scope.go:117] "RemoveContainer" containerID="5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33" Jul 24 14:27:27.322091 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.321805 2568 scope.go:117] "RemoveContainer" containerID="5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33" Jul 24 14:27:27.322391 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:27:27.322142 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33\": container with ID starting with 5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33 not found: ID does not exist" containerID="5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33" Jul 24 14:27:27.322391 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.322164 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33"} err="failed to get container status \"5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33\": rpc error: code = NotFound desc = could not find container \"5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33\": container with ID starting with 5399e2eee9492fcf58775917a13ecc9becd00627f5dc9833d8a5bc59b699bd33 not found: ID does not exist" Jul 24 14:27:27.326830 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.326796 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg"] Jul 24 14:27:27.332645 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.332616 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d95e17-7c7c6c965-n5chg"] Jul 24 14:27:27.344012 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.343979 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p"] Jul 24 14:27:27.348672 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.348639 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d95e17-78db86cd6b-gnp2p"] Jul 24 14:27:27.861394 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.861358 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f9af35-e075-4687-b43f-e511a8a4d774" path="/var/lib/kubelet/pods/53f9af35-e075-4687-b43f-e511a8a4d774/volumes" Jul 24 14:27:27.861896 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:27.861875 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e529802e-2728-40fa-839f-85e8a113ba55" path="/var/lib/kubelet/pods/e529802e-2728-40fa-839f-85e8a113ba55/volumes" Jul 24 14:27:30.808555 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:30.808510 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt"] Jul 24 14:27:30.817510 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:30.817476 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d95e17-vw5kt"] Jul 24 14:27:31.861323 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:31.861287 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fd07165-8665-4500-9e44-cf56a7cbad74" path="/var/lib/kubelet/pods/2fd07165-8665-4500-9e44-cf56a7cbad74/volumes" Jul 24 14:27:41.352816 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.352779 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd"] Jul 24 14:27:41.353253 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353237 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f9af35-e075-4687-b43f-e511a8a4d774" containerName="payload-processing" Jul 24 14:27:41.353304 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353254 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f9af35-e075-4687-b43f-e511a8a4d774" containerName="payload-processing" Jul 24 14:27:41.353304 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353267 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e529802e-2728-40fa-839f-85e8a113ba55" containerName="maas-api" Jul 24 14:27:41.353304 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353273 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e529802e-2728-40fa-839f-85e8a113ba55" containerName="maas-api" Jul 24 14:27:41.353304 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353288 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd07165-8665-4500-9e44-cf56a7cbad74" containerName="revoke-keys" Jul 24 14:27:41.353304 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353294 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd07165-8665-4500-9e44-cf56a7cbad74" containerName="revoke-keys" Jul 24 14:27:41.353468 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353363 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f9af35-e075-4687-b43f-e511a8a4d774" containerName="payload-processing" Jul 24 14:27:41.353468 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353372 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd07165-8665-4500-9e44-cf56a7cbad74" containerName="revoke-keys" Jul 24 14:27:41.353468 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.353380 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e529802e-2728-40fa-839f-85e8a113ba55" containerName="maas-api" Jul 24 14:27:41.356547 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.356531 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:41.359127 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.359100 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:27:41.365063 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.364914 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd"] Jul 24 14:27:41.444159 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.444116 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8acd04-63c7-4eec-b86f-0b9c0974e397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd\" (UID: \"6d8acd04-63c7-4eec-b86f-0b9c0974e397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:41.544694 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.544649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8acd04-63c7-4eec-b86f-0b9c0974e397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd\" (UID: \"6d8acd04-63c7-4eec-b86f-0b9c0974e397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:41.545306 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.545274 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8acd04-63c7-4eec-b86f-0b9c0974e397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd\" (UID: \"6d8acd04-63c7-4eec-b86f-0b9c0974e397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:41.670822 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.670781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:41.818247 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:41.818216 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd"] Jul 24 14:27:41.822058 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:27:41.821992 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8acd04_63c7_4eec_b86f_0b9c0974e397.slice/crio-8b7b0c6b8035714feb7f7d3612f34e1aec861e79ef47d557bba435d972d9bb11 WatchSource:0}: Error finding container 8b7b0c6b8035714feb7f7d3612f34e1aec861e79ef47d557bba435d972d9bb11: Status 404 returned error can't find the container with id 8b7b0c6b8035714feb7f7d3612f34e1aec861e79ef47d557bba435d972d9bb11 Jul 24 14:27:42.374132 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:42.374097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" event={"ID":"6d8acd04-63c7-4eec-b86f-0b9c0974e397","Type":"ContainerStarted","Data":"a75f664122fc4efcf962e3bf698499807ec92522b6957a6d42ea8fccb3b2402f"} Jul 24 14:27:42.374534 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:42.374152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" event={"ID":"6d8acd04-63c7-4eec-b86f-0b9c0974e397","Type":"ContainerStarted","Data":"8b7b0c6b8035714feb7f7d3612f34e1aec861e79ef47d557bba435d972d9bb11"} Jul 24 14:27:42.392608 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:42.392550 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" podStartSLOduration=1.392529344 podStartE2EDuration="1.392529344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:27:42.390522522 +0000 UTC m=+2959.001002270" watchObservedRunningTime="2026-07-24 14:27:42.392529344 +0000 UTC m=+2959.003009105" Jul 24 14:27:43.381653 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:43.381613 2568 generic.go:356] "Generic (PLEG): container finished" podID="6d8acd04-63c7-4eec-b86f-0b9c0974e397" containerID="a75f664122fc4efcf962e3bf698499807ec92522b6957a6d42ea8fccb3b2402f" exitCode=0 Jul 24 14:27:43.382195 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:43.381707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" event={"ID":"6d8acd04-63c7-4eec-b86f-0b9c0974e397","Type":"ContainerDied","Data":"a75f664122fc4efcf962e3bf698499807ec92522b6957a6d42ea8fccb3b2402f"} Jul 24 14:27:44.527504 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:44.527477 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:44.572893 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:44.572861 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8acd04-63c7-4eec-b86f-0b9c0974e397-maas-api-service-ca\") pod \"6d8acd04-63c7-4eec-b86f-0b9c0974e397\" (UID: \"6d8acd04-63c7-4eec-b86f-0b9c0974e397\") " Jul 24 14:27:44.573279 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:44.573254 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8acd04-63c7-4eec-b86f-0b9c0974e397-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d8acd04-63c7-4eec-b86f-0b9c0974e397" (UID: "6d8acd04-63c7-4eec-b86f-0b9c0974e397"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:44.673908 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:44.673824 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d8acd04-63c7-4eec-b86f-0b9c0974e397-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:45.394292 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:45.394261 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" Jul 24 14:27:45.394466 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:45.394299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd" event={"ID":"6d8acd04-63c7-4eec-b86f-0b9c0974e397","Type":"ContainerDied","Data":"8b7b0c6b8035714feb7f7d3612f34e1aec861e79ef47d557bba435d972d9bb11"} Jul 24 14:27:45.394466 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:45.394348 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7b0c6b8035714feb7f7d3612f34e1aec861e79ef47d557bba435d972d9bb11" Jul 24 14:27:56.965631 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:56.965592 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h"] Jul 24 14:27:56.966144 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:56.965813 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" podUID="0757972a-a9db-418e-9b27-756823cbf84d" containerName="payload-processing" containerID="cri-o://a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94" gracePeriod=30 Jul 24 14:27:56.978209 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:56.978177 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq"] Jul 24 14:27:56.978517 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:56.978487 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" podUID="0d3993a2-1937-4dc1-a193-150b2acf3976" containerName="payload-pre-processing" containerID="cri-o://bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74" gracePeriod=30 Jul 24 14:27:57.243360 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.243332 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:27:57.275231 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.275201 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:27:57.296314 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.296282 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0757972a-a9db-418e-9b27-756823cbf84d-plugins-config-volume\") pod \"0757972a-a9db-418e-9b27-756823cbf84d\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " Jul 24 14:27:57.296496 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.296347 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs4lc\" (UniqueName: \"kubernetes.io/projected/0757972a-a9db-418e-9b27-756823cbf84d-kube-api-access-xs4lc\") pod \"0757972a-a9db-418e-9b27-756823cbf84d\" (UID: \"0757972a-a9db-418e-9b27-756823cbf84d\") " Jul 24 14:27:57.296745 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.296695 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0757972a-a9db-418e-9b27-756823cbf84d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0757972a-a9db-418e-9b27-756823cbf84d" (UID: "0757972a-a9db-418e-9b27-756823cbf84d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:57.298866 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.298836 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0757972a-a9db-418e-9b27-756823cbf84d-kube-api-access-xs4lc" (OuterVolumeSpecName: "kube-api-access-xs4lc") pod "0757972a-a9db-418e-9b27-756823cbf84d" (UID: "0757972a-a9db-418e-9b27-756823cbf84d"). InnerVolumeSpecName "kube-api-access-xs4lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:57.397430 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.397394 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3993a2-1937-4dc1-a193-150b2acf3976-plugins-config-volume\") pod \"0d3993a2-1937-4dc1-a193-150b2acf3976\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " Jul 24 14:27:57.397609 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.397445 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrdbw\" (UniqueName: \"kubernetes.io/projected/0d3993a2-1937-4dc1-a193-150b2acf3976-kube-api-access-wrdbw\") pod \"0d3993a2-1937-4dc1-a193-150b2acf3976\" (UID: \"0d3993a2-1937-4dc1-a193-150b2acf3976\") " Jul 24 14:27:57.397868 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.397844 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0757972a-a9db-418e-9b27-756823cbf84d-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.397920 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.397881 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs4lc\" (UniqueName: \"kubernetes.io/projected/0757972a-a9db-418e-9b27-756823cbf84d-kube-api-access-xs4lc\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.397920 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.397841 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3993a2-1937-4dc1-a193-150b2acf3976-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d3993a2-1937-4dc1-a193-150b2acf3976" (UID: "0d3993a2-1937-4dc1-a193-150b2acf3976"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:27:57.399627 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.399604 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3993a2-1937-4dc1-a193-150b2acf3976-kube-api-access-wrdbw" (OuterVolumeSpecName: "kube-api-access-wrdbw") pod "0d3993a2-1937-4dc1-a193-150b2acf3976" (UID: "0d3993a2-1937-4dc1-a193-150b2acf3976"). InnerVolumeSpecName "kube-api-access-wrdbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:27:57.466083 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.466046 2568 generic.go:356] "Generic (PLEG): container finished" podID="0d3993a2-1937-4dc1-a193-150b2acf3976" containerID="bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74" exitCode=0 Jul 24 14:27:57.466264 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.466089 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" Jul 24 14:27:57.466264 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.466141 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" event={"ID":"0d3993a2-1937-4dc1-a193-150b2acf3976","Type":"ContainerDied","Data":"bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74"} Jul 24 14:27:57.466264 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.466202 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq" event={"ID":"0d3993a2-1937-4dc1-a193-150b2acf3976","Type":"ContainerDied","Data":"55866040ef58b03c8983011090d44b3304eac1ef963cd34229f6f85462bc38ae"} Jul 24 14:27:57.466264 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.466234 2568 scope.go:117] "RemoveContainer" containerID="bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74" Jul 24 14:27:57.467673 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.467647 2568 generic.go:356] "Generic (PLEG): container finished" podID="0757972a-a9db-418e-9b27-756823cbf84d" containerID="a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94" exitCode=0 Jul 24 14:27:57.467848 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.467686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" event={"ID":"0757972a-a9db-418e-9b27-756823cbf84d","Type":"ContainerDied","Data":"a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94"} Jul 24 14:27:57.467848 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.467706 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" Jul 24 14:27:57.467848 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.467758 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h" event={"ID":"0757972a-a9db-418e-9b27-756823cbf84d","Type":"ContainerDied","Data":"a023029a886b9ed06015cbf02cb838f638584c00bfeb2d6c7c56e974fd7418bc"} Jul 24 14:27:57.477312 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.477291 2568 scope.go:117] "RemoveContainer" containerID="bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74" Jul 24 14:27:57.477663 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:27:57.477636 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74\": container with ID starting with bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74 not found: ID does not exist" containerID="bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74" Jul 24 14:27:57.477780 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.477673 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74"} err="failed to get container status \"bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74\": rpc error: code = NotFound desc = could not find container \"bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74\": container with ID starting with bf28eced5987d792f1f202bc0566fa298f476312f2be00a62f2d4275031a0b74 not found: ID does not exist" Jul 24 14:27:57.477780 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.477694 2568 scope.go:117] "RemoveContainer" containerID="a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94" Jul 24 14:27:57.486636 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.486615 2568 scope.go:117] "RemoveContainer" containerID="a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94" Jul 24 14:27:57.486937 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:27:57.486919 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94\": container with ID starting with a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94 not found: ID does not exist" containerID="a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94" Jul 24 14:27:57.486994 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.486945 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94"} err="failed to get container status \"a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94\": rpc error: code = NotFound desc = could not find container \"a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94\": container with ID starting with a914e64cee56cb260f3335ab03947cd213a9ccb2e8deae3e2a44331ed53adb94 not found: ID does not exist" Jul 24 14:27:57.497229 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.497165 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h"] Jul 24 14:27:57.498415 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.498389 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrdbw\" (UniqueName: \"kubernetes.io/projected/0d3993a2-1937-4dc1-a193-150b2acf3976-kube-api-access-wrdbw\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.498415 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.498415 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d3993a2-1937-4dc1-a193-150b2acf3976-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:27:57.499938 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.499917 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-15e86b-6d784b4f9b-7cp9h"] Jul 24 14:27:57.515508 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.515477 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq"] Jul 24 14:27:57.524621 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.524595 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-15e86b-74b6f6f6c6-2h2pq"] Jul 24 14:27:57.860635 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.860544 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0757972a-a9db-418e-9b27-756823cbf84d" path="/var/lib/kubelet/pods/0757972a-a9db-418e-9b27-756823cbf84d/volumes" Jul 24 14:27:57.861047 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:27:57.861024 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3993a2-1937-4dc1-a193-150b2acf3976" path="/var/lib/kubelet/pods/0d3993a2-1937-4dc1-a193-150b2acf3976/volumes" Jul 24 14:28:01.436481 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:01.436437 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd"] Jul 24 14:28:01.439053 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:01.439016 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-15e86b-jrhcd"] Jul 24 14:28:01.863772 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:01.863675 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8acd04-63c7-4eec-b86f-0b9c0974e397" path="/var/lib/kubelet/pods/6d8acd04-63c7-4eec-b86f-0b9c0974e397/volumes" Jul 24 14:28:50.689148 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689112 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x"] Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689518 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3993a2-1937-4dc1-a193-150b2acf3976" containerName="payload-pre-processing" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689529 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3993a2-1937-4dc1-a193-150b2acf3976" containerName="payload-pre-processing" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689542 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0757972a-a9db-418e-9b27-756823cbf84d" containerName="payload-processing" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689548 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0757972a-a9db-418e-9b27-756823cbf84d" containerName="payload-processing" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689559 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8acd04-63c7-4eec-b86f-0b9c0974e397" containerName="revoke-keys" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689565 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8acd04-63c7-4eec-b86f-0b9c0974e397" containerName="revoke-keys" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689620 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3993a2-1937-4dc1-a193-150b2acf3976" containerName="payload-pre-processing" Jul 24 14:28:50.689626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689630 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8acd04-63c7-4eec-b86f-0b9c0974e397" containerName="revoke-keys" Jul 24 14:28:50.689943 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.689636 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0757972a-a9db-418e-9b27-756823cbf84d" containerName="payload-processing" Jul 24 14:28:50.692859 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.692842 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:50.695520 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.695498 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:28:50.701269 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.701245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x"] Jul 24 14:28:50.830662 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.830626 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x\" (UID: \"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:50.931217 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.931182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x\" (UID: \"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:50.931867 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:50.931845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x\" (UID: \"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:51.005598 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:51.005505 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:51.153004 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:51.151397 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x"] Jul 24 14:28:51.787984 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:51.787895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" event={"ID":"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f","Type":"ContainerStarted","Data":"27c1f42d49ff672727ae0b3f6e7b7b2e9728c0935a999dec08bd3c712b48ea83"} Jul 24 14:28:51.787984 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:51.787948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" event={"ID":"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f","Type":"ContainerStarted","Data":"8557f51b61c66c04617bcadd31a5b8df771d094cb0fa8ef60b8a717ad3507a88"} Jul 24 14:28:51.804908 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:51.804857 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" podStartSLOduration=1.804840801 podStartE2EDuration="1.804840801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:28:51.802199594 +0000 UTC m=+3028.412679340" watchObservedRunningTime="2026-07-24 14:28:51.804840801 +0000 UTC m=+3028.415320546" Jul 24 14:28:52.795587 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:52.795551 2568 generic.go:356] "Generic (PLEG): container finished" podID="0063b9df-6dbf-4fe3-a7b2-441e00e4a39f" containerID="27c1f42d49ff672727ae0b3f6e7b7b2e9728c0935a999dec08bd3c712b48ea83" exitCode=0 Jul 24 14:28:52.795999 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:52.795627 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" event={"ID":"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f","Type":"ContainerDied","Data":"27c1f42d49ff672727ae0b3f6e7b7b2e9728c0935a999dec08bd3c712b48ea83"} Jul 24 14:28:53.938747 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:53.938686 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:54.060086 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:54.059997 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f-maas-api-service-ca\") pod \"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f\" (UID: \"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f\") " Jul 24 14:28:54.060354 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:54.060330 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0063b9df-6dbf-4fe3-a7b2-441e00e4a39f" (UID: "0063b9df-6dbf-4fe3-a7b2-441e00e4a39f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:28:54.160995 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:54.160955 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:28:54.807749 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:54.807699 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" Jul 24 14:28:54.807926 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:54.807745 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x" event={"ID":"0063b9df-6dbf-4fe3-a7b2-441e00e4a39f","Type":"ContainerDied","Data":"8557f51b61c66c04617bcadd31a5b8df771d094cb0fa8ef60b8a717ad3507a88"} Jul 24 14:28:54.807926 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:28:54.807789 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8557f51b61c66c04617bcadd31a5b8df771d094cb0fa8ef60b8a717ad3507a88" Jul 24 14:29:01.259928 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.259891 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc"] Jul 24 14:29:01.260383 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.260146 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" podUID="2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" containerName="maas-api" containerID="cri-o://915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27" gracePeriod=30 Jul 24 14:29:01.518195 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.518124 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:29:01.531897 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.531863 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-openshift-service-ca\") pod \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " Jul 24 14:29:01.532062 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.531935 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2shwf\" (UniqueName: \"kubernetes.io/projected/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-kube-api-access-2shwf\") pod \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " Jul 24 14:29:01.532062 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.532004 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls\") pod \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\" (UID: \"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4\") " Jul 24 14:29:01.532312 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.532285 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" (UID: "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:01.534143 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.534119 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" (UID: "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 14:29:01.534211 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.534191 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-kube-api-access-2shwf" (OuterVolumeSpecName: "kube-api-access-2shwf") pod "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" (UID: "2a814ae4-15a6-4cab-9b29-fedc0bb58bb4"). InnerVolumeSpecName "kube-api-access-2shwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:01.633111 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.633075 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-maas-api-tls\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.633111 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.633109 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-openshift-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.633111 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.633120 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2shwf\" (UniqueName: \"kubernetes.io/projected/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4-kube-api-access-2shwf\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:01.848307 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.848223 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" containerID="915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27" exitCode=0 Jul 24 14:29:01.848307 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.848274 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" Jul 24 14:29:01.848492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.848308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" event={"ID":"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4","Type":"ContainerDied","Data":"915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27"} Jul 24 14:29:01.848492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.848369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc" event={"ID":"2a814ae4-15a6-4cab-9b29-fedc0bb58bb4","Type":"ContainerDied","Data":"8c7f0cad2ae2393f0413d5050f50286cb389ef0dc778fe522baaed38b22b77d8"} Jul 24 14:29:01.848492 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.848398 2568 scope.go:117] "RemoveContainer" containerID="915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27" Jul 24 14:29:01.859230 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.859200 2568 scope.go:117] "RemoveContainer" containerID="915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27" Jul 24 14:29:01.859486 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:29:01.859468 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27\": container with ID starting with 915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27 not found: ID does not exist" containerID="915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27" Jul 24 14:29:01.859550 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.859494 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27"} err="failed to get container status \"915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27\": rpc error: code = NotFound desc = could not find container \"915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27\": container with ID starting with 915fd0106e82ac146135d84630f27a6666044ee99246711f261573a0c4afcc27 not found: ID does not exist" Jul 24 14:29:01.873379 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.873345 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc"] Jul 24 14:29:01.877521 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:01.877490 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03c962-7f8b5fb4b7-5v9bc"] Jul 24 14:29:03.860586 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:03.860546 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" path="/var/lib/kubelet/pods/2a814ae4-15a6-4cab-9b29-fedc0bb58bb4/volumes" Jul 24 14:29:05.774623 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:05.774589 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x"] Jul 24 14:29:05.777175 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:05.777150 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03c962-4g55x"] Jul 24 14:29:05.860929 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:05.860894 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0063b9df-6dbf-4fe3-a7b2-441e00e4a39f" path="/var/lib/kubelet/pods/0063b9df-6dbf-4fe3-a7b2-441e00e4a39f/volumes" Jul 24 14:29:13.009955 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.009912 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9"] Jul 24 14:29:13.010457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.010309 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" containerName="maas-api" Jul 24 14:29:13.010457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.010321 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" containerName="maas-api" Jul 24 14:29:13.010457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.010333 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0063b9df-6dbf-4fe3-a7b2-441e00e4a39f" containerName="revoke-keys" Jul 24 14:29:13.010457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.010338 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0063b9df-6dbf-4fe3-a7b2-441e00e4a39f" containerName="revoke-keys" Jul 24 14:29:13.010457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.010406 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a814ae4-15a6-4cab-9b29-fedc0bb58bb4" containerName="maas-api" Jul 24 14:29:13.010457 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.010414 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0063b9df-6dbf-4fe3-a7b2-441e00e4a39f" containerName="revoke-keys" Jul 24 14:29:13.013823 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.013805 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:13.017324 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.017300 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:29:13.027146 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.027113 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9"] Jul 24 14:29:13.144646 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.144613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9\" (UID: \"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:13.245787 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.245715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9\" (UID: \"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:13.246372 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.246351 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9\" (UID: \"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:13.324645 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.324538 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:13.471375 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.471343 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9"] Jul 24 14:29:13.473318 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:29:13.473290 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7608f75a_1fd6_4c2c_bfc9_364bbed5a3d1.slice/crio-ffc0457ca7b0e43a18f5e871cd0cd332687fd77c188c713249be36b9862b739d WatchSource:0}: Error finding container ffc0457ca7b0e43a18f5e871cd0cd332687fd77c188c713249be36b9862b739d: Status 404 returned error can't find the container with id ffc0457ca7b0e43a18f5e871cd0cd332687fd77c188c713249be36b9862b739d Jul 24 14:29:13.926936 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.926893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" event={"ID":"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1","Type":"ContainerStarted","Data":"e0d287371e870d8f06e54bbc4869d07e6543c33f3021f7be9ba12a995b1d1179"} Jul 24 14:29:13.927142 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.926951 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" event={"ID":"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1","Type":"ContainerStarted","Data":"ffc0457ca7b0e43a18f5e871cd0cd332687fd77c188c713249be36b9862b739d"} Jul 24 14:29:13.944891 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:13.944760 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" podStartSLOduration=1.944744764 podStartE2EDuration="1.944744764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:29:13.943420366 +0000 UTC m=+3050.553900109" watchObservedRunningTime="2026-07-24 14:29:13.944744764 +0000 UTC m=+3050.555224501" Jul 24 14:29:14.934042 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:14.934008 2568 generic.go:356] "Generic (PLEG): container finished" podID="7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1" containerID="e0d287371e870d8f06e54bbc4869d07e6543c33f3021f7be9ba12a995b1d1179" exitCode=0 Jul 24 14:29:14.934459 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:14.934057 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" event={"ID":"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1","Type":"ContainerDied","Data":"e0d287371e870d8f06e54bbc4869d07e6543c33f3021f7be9ba12a995b1d1179"} Jul 24 14:29:16.081256 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.081228 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:16.172260 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.172216 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1-maas-api-service-ca\") pod \"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1\" (UID: \"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1\") " Jul 24 14:29:16.172599 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.172574 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1" (UID: "7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:16.273868 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.273773 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:16.946324 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.946293 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" Jul 24 14:29:16.946501 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.946292 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9" event={"ID":"7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1","Type":"ContainerDied","Data":"ffc0457ca7b0e43a18f5e871cd0cd332687fd77c188c713249be36b9862b739d"} Jul 24 14:29:16.946501 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:16.946418 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc0457ca7b0e43a18f5e871cd0cd332687fd77c188c713249be36b9862b739d" Jul 24 14:29:23.525758 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.525704 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg"] Jul 24 14:29:23.526285 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.525978 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" podUID="46afc43a-1618-4d36-bc69-3540964b5783" containerName="payload-pre-processing" containerID="cri-o://5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b" gracePeriod=30 Jul 24 14:29:23.790910 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.790843 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:29:23.850553 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.850511 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46afc43a-1618-4d36-bc69-3540964b5783-plugins-config-volume\") pod \"46afc43a-1618-4d36-bc69-3540964b5783\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " Jul 24 14:29:23.850779 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.850633 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvmvh\" (UniqueName: \"kubernetes.io/projected/46afc43a-1618-4d36-bc69-3540964b5783-kube-api-access-zvmvh\") pod \"46afc43a-1618-4d36-bc69-3540964b5783\" (UID: \"46afc43a-1618-4d36-bc69-3540964b5783\") " Jul 24 14:29:23.851006 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.850982 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46afc43a-1618-4d36-bc69-3540964b5783-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46afc43a-1618-4d36-bc69-3540964b5783" (UID: "46afc43a-1618-4d36-bc69-3540964b5783"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:23.852807 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.852777 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46afc43a-1618-4d36-bc69-3540964b5783-kube-api-access-zvmvh" (OuterVolumeSpecName: "kube-api-access-zvmvh") pod "46afc43a-1618-4d36-bc69-3540964b5783" (UID: "46afc43a-1618-4d36-bc69-3540964b5783"). InnerVolumeSpecName "kube-api-access-zvmvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:23.952176 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.952133 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46afc43a-1618-4d36-bc69-3540964b5783-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:23.952176 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.952166 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmvh\" (UniqueName: \"kubernetes.io/projected/46afc43a-1618-4d36-bc69-3540964b5783-kube-api-access-zvmvh\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:23.987617 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.987583 2568 generic.go:356] "Generic (PLEG): container finished" podID="46afc43a-1618-4d36-bc69-3540964b5783" containerID="5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b" exitCode=0 Jul 24 14:29:23.987803 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.987629 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" Jul 24 14:29:23.987803 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.987670 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" event={"ID":"46afc43a-1618-4d36-bc69-3540964b5783","Type":"ContainerDied","Data":"5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b"} Jul 24 14:29:23.987803 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.987738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg" event={"ID":"46afc43a-1618-4d36-bc69-3540964b5783","Type":"ContainerDied","Data":"aa38772dd9fdf294a981d652101f58708386b523ef7ea07e3013db93258e29e5"} Jul 24 14:29:23.987803 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.987769 2568 scope.go:117] "RemoveContainer" containerID="5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b" Jul 24 14:29:23.996541 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.996517 2568 scope.go:117] "RemoveContainer" containerID="5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b" Jul 24 14:29:23.996816 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:29:23.996797 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b\": container with ID starting with 5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b not found: ID does not exist" containerID="5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b" Jul 24 14:29:23.996879 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:23.996823 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b"} err="failed to get container status \"5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b\": rpc error: code = NotFound desc = could not find container \"5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b\": container with ID starting with 5ccaa2ed9b5585898ef695949109a912e6b556fceb421fbbea23c3119ebc0b4b not found: ID does not exist" Jul 24 14:29:24.007553 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:24.007516 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg"] Jul 24 14:29:24.012245 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:24.012220 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-a26cce-55c6c447cb-4hzdg"] Jul 24 14:29:25.860332 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:25.860292 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46afc43a-1618-4d36-bc69-3540964b5783" path="/var/lib/kubelet/pods/46afc43a-1618-4d36-bc69-3540964b5783/volumes" Jul 24 14:29:28.100911 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:28.100870 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9"] Jul 24 14:29:28.108145 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:28.108115 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a26cce-z9kq9"] Jul 24 14:29:29.859854 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:29.859820 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1" path="/var/lib/kubelet/pods/7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1/volumes" Jul 24 14:29:35.879875 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.879832 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w"] Jul 24 14:29:35.880440 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.880418 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1" containerName="revoke-keys" Jul 24 14:29:35.880526 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.880443 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1" containerName="revoke-keys" Jul 24 14:29:35.880526 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.880491 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46afc43a-1618-4d36-bc69-3540964b5783" containerName="payload-pre-processing" Jul 24 14:29:35.880526 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.880503 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="46afc43a-1618-4d36-bc69-3540964b5783" containerName="payload-pre-processing" Jul 24 14:29:35.880681 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.880596 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7608f75a-1fd6-4c2c-bfc9-364bbed5a3d1" containerName="revoke-keys" Jul 24 14:29:35.880681 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.880612 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="46afc43a-1618-4d36-bc69-3540964b5783" containerName="payload-pre-processing" Jul 24 14:29:35.883779 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.883757 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:35.887854 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.887835 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dbqcd\"" Jul 24 14:29:35.899628 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:35.899595 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w"] Jul 24 14:29:36.061415 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:36.061377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759f5715-ddfa-43dd-a654-9351f1b11694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w\" (UID: \"759f5715-ddfa-43dd-a654-9351f1b11694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:36.162331 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:36.162240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759f5715-ddfa-43dd-a654-9351f1b11694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w\" (UID: \"759f5715-ddfa-43dd-a654-9351f1b11694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:36.162960 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:36.162942 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759f5715-ddfa-43dd-a654-9351f1b11694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w\" (UID: \"759f5715-ddfa-43dd-a654-9351f1b11694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:36.194401 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:36.194364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:36.344290 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:36.344262 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w"] Jul 24 14:29:36.345800 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:29:36.345769 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759f5715_ddfa_43dd_a654_9351f1b11694.slice/crio-309ef18eceb8b638eebda4a25abf551688cf33736d7c02cd341343d2fc9c1bdd WatchSource:0}: Error finding container 309ef18eceb8b638eebda4a25abf551688cf33736d7c02cd341343d2fc9c1bdd: Status 404 returned error can't find the container with id 309ef18eceb8b638eebda4a25abf551688cf33736d7c02cd341343d2fc9c1bdd Jul 24 14:29:37.065163 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:37.065120 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" event={"ID":"759f5715-ddfa-43dd-a654-9351f1b11694","Type":"ContainerStarted","Data":"42d6bd5db6c2638a999ba94e2531a2d91ab177958c93956f291c3c5c1669921f"} Jul 24 14:29:37.065577 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:37.065191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" event={"ID":"759f5715-ddfa-43dd-a654-9351f1b11694","Type":"ContainerStarted","Data":"309ef18eceb8b638eebda4a25abf551688cf33736d7c02cd341343d2fc9c1bdd"} Jul 24 14:29:37.086106 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:37.086057 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" podStartSLOduration=2.086039941 podStartE2EDuration="2.086039941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 14:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:29:37.085180376 +0000 UTC m=+3073.695660122" watchObservedRunningTime="2026-07-24 14:29:37.086039941 +0000 UTC m=+3073.696519686" Jul 24 14:29:38.072177 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:38.072143 2568 generic.go:356] "Generic (PLEG): container finished" podID="759f5715-ddfa-43dd-a654-9351f1b11694" containerID="42d6bd5db6c2638a999ba94e2531a2d91ab177958c93956f291c3c5c1669921f" exitCode=0 Jul 24 14:29:38.072626 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:38.072184 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" event={"ID":"759f5715-ddfa-43dd-a654-9351f1b11694","Type":"ContainerDied","Data":"42d6bd5db6c2638a999ba94e2531a2d91ab177958c93956f291c3c5c1669921f"} Jul 24 14:29:39.221689 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:39.221661 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:39.395407 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:39.395360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759f5715-ddfa-43dd-a654-9351f1b11694-maas-api-service-ca\") pod \"759f5715-ddfa-43dd-a654-9351f1b11694\" (UID: \"759f5715-ddfa-43dd-a654-9351f1b11694\") " Jul 24 14:29:39.395788 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:39.395764 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759f5715-ddfa-43dd-a654-9351f1b11694-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "759f5715-ddfa-43dd-a654-9351f1b11694" (UID: "759f5715-ddfa-43dd-a654-9351f1b11694"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:39.496466 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:39.496433 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/759f5715-ddfa-43dd-a654-9351f1b11694-maas-api-service-ca\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:40.089370 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:40.089339 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" Jul 24 14:29:40.089545 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:40.089369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w" event={"ID":"759f5715-ddfa-43dd-a654-9351f1b11694","Type":"ContainerDied","Data":"309ef18eceb8b638eebda4a25abf551688cf33736d7c02cd341343d2fc9c1bdd"} Jul 24 14:29:40.089545 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:40.089442 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309ef18eceb8b638eebda4a25abf551688cf33736d7c02cd341343d2fc9c1bdd" Jul 24 14:29:46.357975 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.357936 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds"] Jul 24 14:29:46.358951 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.358884 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" podUID="11a139cb-afad-4ec8-b567-541387bbe0c1" containerName="payload-processing" containerID="cri-o://6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7" gracePeriod=30 Jul 24 14:29:46.626957 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.626923 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:29:46.661346 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.661314 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a139cb-afad-4ec8-b567-541387bbe0c1-plugins-config-volume\") pod \"11a139cb-afad-4ec8-b567-541387bbe0c1\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " Jul 24 14:29:46.661503 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.661360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5lzr\" (UniqueName: \"kubernetes.io/projected/11a139cb-afad-4ec8-b567-541387bbe0c1-kube-api-access-q5lzr\") pod \"11a139cb-afad-4ec8-b567-541387bbe0c1\" (UID: \"11a139cb-afad-4ec8-b567-541387bbe0c1\") " Jul 24 14:29:46.661846 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.661815 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a139cb-afad-4ec8-b567-541387bbe0c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11a139cb-afad-4ec8-b567-541387bbe0c1" (UID: "11a139cb-afad-4ec8-b567-541387bbe0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 14:29:46.663570 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.663545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a139cb-afad-4ec8-b567-541387bbe0c1-kube-api-access-q5lzr" (OuterVolumeSpecName: "kube-api-access-q5lzr") pod "11a139cb-afad-4ec8-b567-541387bbe0c1" (UID: "11a139cb-afad-4ec8-b567-541387bbe0c1"). InnerVolumeSpecName "kube-api-access-q5lzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 14:29:46.762140 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.762104 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a139cb-afad-4ec8-b567-541387bbe0c1-plugins-config-volume\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:46.762140 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:46.762139 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lzr\" (UniqueName: \"kubernetes.io/projected/11a139cb-afad-4ec8-b567-541387bbe0c1-kube-api-access-q5lzr\") on node \"ip-10-0-138-248.ec2.internal\" DevicePath \"\"" Jul 24 14:29:47.130508 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.130472 2568 generic.go:356] "Generic (PLEG): container finished" podID="11a139cb-afad-4ec8-b567-541387bbe0c1" containerID="6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7" exitCode=0 Jul 24 14:29:47.130688 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.130515 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" Jul 24 14:29:47.130688 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.130557 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" event={"ID":"11a139cb-afad-4ec8-b567-541387bbe0c1","Type":"ContainerDied","Data":"6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7"} Jul 24 14:29:47.130688 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.130616 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds" event={"ID":"11a139cb-afad-4ec8-b567-541387bbe0c1","Type":"ContainerDied","Data":"2c416e36acbb3c03e43fc53d66c440b84883c4acf00e147f288cee24e91c64c1"} Jul 24 14:29:47.130688 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.130643 2568 scope.go:117] "RemoveContainer" containerID="6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7" Jul 24 14:29:47.141800 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.141667 2568 scope.go:117] "RemoveContainer" containerID="6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7" Jul 24 14:29:47.142870 ip-10-0-138-248 kubenswrapper[2568]: E0724 14:29:47.142842 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7\": container with ID starting with 6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7 not found: ID does not exist" containerID="6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7" Jul 24 14:29:47.142989 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.142881 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7"} err="failed to get container status \"6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7\": rpc error: code = NotFound desc = could not find container \"6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7\": container with ID starting with 6a277b60cafa38a48596057b865158c2f582d08809f86b65b82f05334fac8ac7 not found: ID does not exist" Jul 24 14:29:47.169659 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.169629 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds"] Jul 24 14:29:47.176163 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.176134 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-cb9aa5-68578467f4-wwxds"] Jul 24 14:29:47.859527 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:47.859493 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11a139cb-afad-4ec8-b567-541387bbe0c1" path="/var/lib/kubelet/pods/11a139cb-afad-4ec8-b567-541387bbe0c1/volumes" Jul 24 14:29:50.963153 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:50.963114 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w"] Jul 24 14:29:50.965611 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:50.965581 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-cb9aa5-vj24w"] Jul 24 14:29:51.860284 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:29:51.860249 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759f5715-ddfa-43dd-a654-9351f1b11694" path="/var/lib/kubelet/pods/759f5715-ddfa-43dd-a654-9351f1b11694/volumes" Jul 24 14:30:05.100331 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:05.100300 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v8dpq_eccc1867-2a09-47a2-a98c-8d024a1bd00a/server/0.log" Jul 24 14:30:05.526985 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:05.526958 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-sps8l_dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5/copy-manifests/0.log" Jul 24 14:30:05.538264 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:05.538237 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-sps8l_dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5/manager/0.log" Jul 24 14:30:06.088205 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:06.088172 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-f95b6f759-5k9t6_0559ef72-57d2-44a9-bd4f-2db513596604/maas-api/0.log" Jul 24 14:30:07.454075 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.454044 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw_c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f/util/0.log" Jul 24 14:30:07.465114 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.465086 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw_c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f/pull/0.log" Jul 24 14:30:07.474535 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.474510 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw_c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f/extract/0.log" Jul 24 14:30:07.596059 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.596034 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7_08220aaa-4d1f-406b-b09d-df1f4c22db64/util/0.log" Jul 24 14:30:07.605022 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.604998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7_08220aaa-4d1f-406b-b09d-df1f4c22db64/pull/0.log" Jul 24 14:30:07.616880 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.616856 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7_08220aaa-4d1f-406b-b09d-df1f4c22db64/extract/0.log" Jul 24 14:30:07.743950 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.743876 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph_4750570c-2073-4de1-a505-7ea27b5d7ff3/util/0.log" Jul 24 14:30:07.756161 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.756114 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph_4750570c-2073-4de1-a505-7ea27b5d7ff3/pull/0.log" Jul 24 14:30:07.768510 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.768484 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph_4750570c-2073-4de1-a505-7ea27b5d7ff3/extract/0.log" Jul 24 14:30:07.890918 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.890891 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6_1a8ebba4-8ff0-446a-b60c-bc223a1239ff/util/0.log" Jul 24 14:30:07.903105 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.903077 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6_1a8ebba4-8ff0-446a-b60c-bc223a1239ff/pull/0.log" Jul 24 14:30:07.915015 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:07.914992 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6_1a8ebba4-8ff0-446a-b60c-bc223a1239ff/extract/0.log" Jul 24 14:30:09.434243 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:09.434203 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7vtpl_3699c26b-4e81-4199-a863-2b6e0011639b/discovery/0.log" Jul 24 14:30:10.699944 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:10.699912 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-92sv6_20a99fcd-f54d-404c-8f74-adf733edf234/main/0.log" Jul 24 14:30:11.092405 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:11.092333 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-l2lvm_b66324d1-03e8-4363-ad3b-58bb9e309235/main/0.log" Jul 24 14:30:21.815460 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:21.815423 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vnqvq_8b70c494-5fd3-4783-a114-907bf3f3805e/global-pull-secret-syncer/0.log" Jul 24 14:30:21.908913 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:21.908881 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sw9nh_1eb7472c-bbd5-44be-9952-53f299a387e9/konnectivity-agent/0.log" Jul 24 14:30:21.967744 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:21.967706 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-248.ec2.internal_771381fabc06c5e919c1029bae495d3d/haproxy/0.log" Jul 24 14:30:26.684140 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.684110 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw_c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f/extract/0.log" Jul 24 14:30:26.723343 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.723316 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw_c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f/util/0.log" Jul 24 14:30:26.754102 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.754069 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759rmrcw_c42e23d2-3ef2-4dc1-ba3e-9aa8c4371c8f/pull/0.log" Jul 24 14:30:26.787163 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.787134 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7_08220aaa-4d1f-406b-b09d-df1f4c22db64/extract/0.log" Jul 24 14:30:26.815328 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.815302 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7_08220aaa-4d1f-406b-b09d-df1f4c22db64/util/0.log" Jul 24 14:30:26.843485 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.843455 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0mg6j7_08220aaa-4d1f-406b-b09d-df1f4c22db64/pull/0.log" Jul 24 14:30:26.888607 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.888580 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph_4750570c-2073-4de1-a505-7ea27b5d7ff3/extract/0.log" Jul 24 14:30:26.915924 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.915888 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph_4750570c-2073-4de1-a505-7ea27b5d7ff3/util/0.log" Jul 24 14:30:26.944088 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.944007 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fqzph_4750570c-2073-4de1-a505-7ea27b5d7ff3/pull/0.log" Jul 24 14:30:26.973597 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:26.973568 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6_1a8ebba4-8ff0-446a-b60c-bc223a1239ff/extract/0.log" Jul 24 14:30:27.005691 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:27.005662 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6_1a8ebba4-8ff0-446a-b60c-bc223a1239ff/util/0.log" Jul 24 14:30:27.029543 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:27.029517 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gvnh6_1a8ebba4-8ff0-446a-b60c-bc223a1239ff/pull/0.log" Jul 24 14:30:29.679319 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.679293 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/alertmanager/0.log" Jul 24 14:30:29.708615 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.708583 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/config-reloader/0.log" Jul 24 14:30:29.734841 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.734813 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/kube-rbac-proxy-web/0.log" Jul 24 14:30:29.760221 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.760140 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/kube-rbac-proxy/0.log" Jul 24 14:30:29.784104 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.784065 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/kube-rbac-proxy-metric/0.log" Jul 24 14:30:29.811283 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.811252 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/prom-label-proxy/0.log" Jul 24 14:30:29.837471 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.837439 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd9f2df5-bd7d-43c9-8751-0e3ced766e3f/init-config-reloader/0.log" Jul 24 14:30:29.882400 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:29.882358 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-kqlvx_49a30465-6733-4640-b754-da43d4c2c0b2/cluster-monitoring-operator/0.log" Jul 24 14:30:30.010074 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.010040 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67495bb879-g42z7_abb93816-b770-4025-94da-7bc8bbe942dd/metrics-server/0.log" Jul 24 14:30:30.039967 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.039880 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-lx9g9_d688c9f8-373a-465a-98c3-76f1d9488f74/monitoring-plugin/0.log" Jul 24 14:30:30.075847 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.075818 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666fz_79570920-ef63-4977-a3ab-01c8047f01a4/node-exporter/0.log" Jul 24 14:30:30.098189 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.098161 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666fz_79570920-ef63-4977-a3ab-01c8047f01a4/kube-rbac-proxy/0.log" Jul 24 14:30:30.125132 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.125109 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-666fz_79570920-ef63-4977-a3ab-01c8047f01a4/init-textfile/0.log" Jul 24 14:30:30.799733 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.799635 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-542dq_0d43f1ef-1489-44a6-b5f8-ea4ae26dc687/prometheus-operator-admission-webhook/0.log" Jul 24 14:30:30.834834 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.834786 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fb74759f-dfw7t_4421bf75-f6c0-4329-96d6-3ac88c0fa63b/telemeter-client/0.log" Jul 24 14:30:30.860872 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.860840 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fb74759f-dfw7t_4421bf75-f6c0-4329-96d6-3ac88c0fa63b/reload/0.log" Jul 24 14:30:30.886302 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:30.886273 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fb74759f-dfw7t_4421bf75-f6c0-4329-96d6-3ac88c0fa63b/kube-rbac-proxy/0.log" Jul 24 14:30:32.284142 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.284110 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-bcjmn_883ffe42-65bc-4a6f-8f14-d4285bc94b90/networking-console-plugin/0.log" Jul 24 14:30:32.538576 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.538490 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5"] Jul 24 14:30:32.538918 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.538905 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759f5715-ddfa-43dd-a654-9351f1b11694" containerName="revoke-keys" Jul 24 14:30:32.538982 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.538920 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="759f5715-ddfa-43dd-a654-9351f1b11694" containerName="revoke-keys" Jul 24 14:30:32.538982 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.538929 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a139cb-afad-4ec8-b567-541387bbe0c1" containerName="payload-processing" Jul 24 14:30:32.538982 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.538936 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a139cb-afad-4ec8-b567-541387bbe0c1" containerName="payload-processing" Jul 24 14:30:32.539085 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.539000 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="759f5715-ddfa-43dd-a654-9351f1b11694" containerName="revoke-keys" Jul 24 14:30:32.539085 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.539011 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="11a139cb-afad-4ec8-b567-541387bbe0c1" containerName="payload-processing" Jul 24 14:30:32.543296 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.543276 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.546386 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.546363 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njvgn\"/\"kube-root-ca.crt\"" Jul 24 14:30:32.547333 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.547317 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-njvgn\"/\"openshift-service-ca.crt\"" Jul 24 14:30:32.547406 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.547317 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-njvgn\"/\"default-dockercfg-bdtjz\"" Jul 24 14:30:32.553558 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.553532 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5"] Jul 24 14:30:32.609813 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.609770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-proc\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.610017 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.609837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-sys\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.610017 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.609900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-lib-modules\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.610017 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.609961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-podres\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.610017 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.609991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vbg\" (UniqueName: \"kubernetes.io/projected/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-kube-api-access-l2vbg\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711425 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-sys\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711425 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-lib-modules\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711642 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711451 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-podres\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711642 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vbg\" (UniqueName: \"kubernetes.io/projected/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-kube-api-access-l2vbg\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711642 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711526 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-sys\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711642 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-proc\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711840 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711635 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-proc\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711840 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-lib-modules\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.711840 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.711598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-podres\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.721836 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.721810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vbg\" (UniqueName: \"kubernetes.io/projected/6ca63bb5-1712-44ce-a295-01bb4f0e5c3a-kube-api-access-l2vbg\") pod \"perf-node-gather-daemonset-tqbc5\" (UID: \"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a\") " pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.854192 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.854082 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:32.970971 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:32.970941 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-cqct5_2254edb8-c574-45db-99a8-320568e891cc/console-operator/0.log" Jul 24 14:30:33.004975 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.004941 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5"] Jul 24 14:30:33.006193 ip-10-0-138-248 kubenswrapper[2568]: W0724 14:30:33.006160 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ca63bb5_1712_44ce_a295_01bb4f0e5c3a.slice/crio-69077b45ec3cd1878cb0ff4b04ac1770c37a0d688a983e24dbd8791dfbdd5683 WatchSource:0}: Error finding container 69077b45ec3cd1878cb0ff4b04ac1770c37a0d688a983e24dbd8791dfbdd5683: Status 404 returned error can't find the container with id 69077b45ec3cd1878cb0ff4b04ac1770c37a0d688a983e24dbd8791dfbdd5683 Jul 24 14:30:33.393908 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.393867 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" event={"ID":"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a","Type":"ContainerStarted","Data":"4195f339b23bb6390b35577b097e74954002bbd5b5bf55014211eac14686eb59"} Jul 24 14:30:33.394300 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.393935 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:33.394300 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.393957 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" event={"ID":"6ca63bb5-1712-44ce-a295-01bb4f0e5c3a","Type":"ContainerStarted","Data":"69077b45ec3cd1878cb0ff4b04ac1770c37a0d688a983e24dbd8791dfbdd5683"} Jul 24 14:30:33.414581 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.414526 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" podStartSLOduration=1.414509407 podStartE2EDuration="1.414509407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 14:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 14:30:33.412352015 +0000 UTC m=+3130.022831755" watchObservedRunningTime="2026-07-24 14:30:33.414509407 +0000 UTC m=+3130.024989152" Jul 24 14:30:33.533872 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.533836 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84f6449495-ntc8w_f12e7f22-8e04-49d3-bed0-60f9a1f145c5/console/0.log" Jul 24 14:30:33.570126 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:33.570092 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-f7znd_6f5612fb-a3c0-4449-8f18-2b4db6d6252d/download-server/0.log" Jul 24 14:30:34.152027 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:34.152001 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-25qsv_22bf929a-f86e-40a9-bf36-2a974ed98596/volume-data-source-validator/0.log" Jul 24 14:30:35.362544 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:35.362515 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k8lk6_d80464ab-57c8-4be8-af66-31519ad6beaa/dns/0.log" Jul 24 14:30:35.387947 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:35.387920 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k8lk6_d80464ab-57c8-4be8-af66-31519ad6beaa/kube-rbac-proxy/0.log" Jul 24 14:30:35.497595 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:35.497563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c5q46_e83441c2-b725-4664-91d0-5e915d2ba6c1/dns-node-resolver/0.log" Jul 24 14:30:36.103892 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:36.103857 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fc58d8d96-zhw59_287e3ad6-10ce-4881-96cd-a179071bacb9/registry/0.log" Jul 24 14:30:36.158896 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:36.158868 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kcsmh_241fa270-01bf-44d5-a857-22457556bd4f/node-ca/0.log" Jul 24 14:30:37.379385 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:37.379345 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7vtpl_3699c26b-4e81-4199-a863-2b6e0011639b/discovery/0.log" Jul 24 14:30:38.432433 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:38.432402 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4g25d_a40269b1-49be-404b-ab9e-c645775f0848/serve-healthcheck-canary/0.log" Jul 24 14:30:39.062980 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:39.062945 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-fhj28_500cd154-214f-47e3-b02e-e5bdeeef3f45/insights-operator/0.log" Jul 24 14:30:39.370155 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:39.370124 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r54hx_10c4847c-e22b-4313-b08d-e0f477a8c33e/kube-rbac-proxy/0.log" Jul 24 14:30:39.409481 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:39.409454 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-njvgn/perf-node-gather-daemonset-tqbc5" Jul 24 14:30:39.415025 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:39.415005 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r54hx_10c4847c-e22b-4313-b08d-e0f477a8c33e/exporter/0.log" Jul 24 14:30:39.457240 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:39.457209 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r54hx_10c4847c-e22b-4313-b08d-e0f477a8c33e/extractor/0.log" Jul 24 14:30:42.477255 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:42.477221 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v8dpq_eccc1867-2a09-47a2-a98c-8d024a1bd00a/server/0.log" Jul 24 14:30:42.766576 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:42.766492 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-sps8l_dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5/manager/0.log" Jul 24 14:30:42.805907 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:42.805878 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c96dc84dd-sps8l_dadf38ad-d85c-4cd7-bbcf-053f7d3df4b5/copy-manifests/0.log" Jul 24 14:30:43.028901 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:43.028804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7vtpl_3699c26b-4e81-4199-a863-2b6e0011639b/discovery/0.log" Jul 24 14:30:44.178641 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:44.178607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7f486947b8-4zz5d_8a465945-501b-4ccb-b96a-16dfeb24755a/manager/0.log" Jul 24 14:30:44.236033 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:44.235998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mmqh8_3926899b-6fb6-4e21-b4a9-31322ea556f7/openshift-lws-operator/0.log" Jul 24 14:30:44.724102 ip-10-0-138-248 kubenswrapper[2568]: I0724 14:30:44.724071 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7vtpl_3699c26b-4e81-4199-a863-2b6e0011639b/discovery/0.log"